python - Huge array representing redundant and sparse data optimization -

-

i tried search answer question, exists couldn't find any, because don't know how name it.

so let's point.

i working on statistical model. dataset contains information 45 public transport stations , static information them (so 45 rows , around 1000 feature columns) , regularly-spaced temporal measurements (so 2 000 000 rows 10 columns). "real" amount of information in small enough (around 500 mb) processed easily.

the problem statistical modules on python require simple 2d arrays, numpy's arrays. have combine data, 2 000 000 rows of measure have attach 1000+ columns of features related station in measure took place end 17 gb database... of redundant , feel waste of resource.

i have idea have absolutely no idea of how : array function given , j returns value, possible me "emulate", or come fake array, pseudo array, array interface, can accepted numpy array modules? don't see why couldn't it, because array function (i:j) -> a[i][j]. access little slower memory access of classic array problem, , still fast. don't know how it...

thank in advance , tell me if need bring more information or change way ask question!

edit :

ok maybe can clarify problem little bit :

my data can presented in relational database or object database fashion. pretty small (500 mb) in format, when merge tables make possible scikit-learn process it, becomes way big (17 gb +) , seems waste of memory! use techniques handle big data, may solution, isn't possible prevent doing so? can use data directly in scikit_learn without having merge explicitly tables? can make implicit merging emulates datastructure without taking additional memory?


Comments

Post a Comment

Popular posts from this blog

java - Static nested class instance -

-
this question has answer here: java inner class , static nested class 23 answers i came across in java, know static nested class why create instance of 'static'. isn't whole idea of 'static' use without instance? understand concept of inner classes, why (and frankly how possible to) create 'instance' of 'static' member @ ? why this: 1- outerclass.staticnestedclass nestedobject = new outerclass.staticnestedclass(); 2- nestedobject.anestedclassmethod(); and not this: 1- outerclass outerinstance=new outerclass(); 2- outerinstance.staticnestedclass.anestedclassmethod(); use on inner classes, keyword static indicates can access inner class without instance of outer class. for example: public class outer { public static class inner { /* code here. */ } } with construction, can create instance of inn
Read more

c# - Bluetooth LE CanUpdate Characteristic property -

-
i building xamarin.forms cross platform mobile app, uses monkey.robotics bluetoth low energy functionality. connecting mbed based implimentation of custom gatt service . in xamarin c#, triggers characteristic. canupdate property in monkey.robotics? this standard example c# based on: if (characteristic.canupdate) { characteristic.valueupdated += (s, e) => { debug.writeline("characteristic.valueupdated"); device.begininvokeonmainthread( () => { updatedisplay(characteristic); }); isbusy = false; // spin until first result received }; isbusy = true; characteristic.startupdates(); } this has been working, since changed own custom gatt service connecting to, canupdate property false. property , how triggered? me debugging gatt service code. thanks monkey.robotics open sour
Read more

JavaScript - Replace variable from string in all occurrences -

-
ok, know if have character '-' , want remove in places in string javascript, ... someword = someword.replace(/-/g, ''); but, when applying array of characters, s not working ... const badchars = ('\/:*?"<>|').split(''); let filename = title.replace(/ /g, '-').tolocalelowercase(); (let item = 0; item < badchars.length; item++) { // below not work global '/ /g' filename = filename.replace(/badchars[item]/g, ''); } any ideas? /badchars[item]/g looks badchars , literally, followed i , t , e , or m . if you're trying use character badchars[item] , you'll need use regexp constructor, , you'll need escape regex-specific characters. escaping regular expression has been well-covered . using that: filename = filename.replace(new regexp(regexp.quote(badchars[item]), 'g'), ''); but , don't want that. want character class: let filename = title.replac
Read more