This is the corrected version:import pandas as pd file_name = r'C:\Users\fazliakhmetovRV\Documents\Python Scripts\primer.xlsx' out = r'C:\Users\fazliakhmetovRV\Documents\Python Scripts\out.xlsx' cols = ['label','x','y','z','value'] df = pd.read_excel(file_name, sheetname='er', skiprows=4, header=None, parse_cols='C:XFD') dfs = [] for i in range(df.columns.size//5): lbl_col = 5*i # filter each 5-column block x = df.ix[(df[lbl_col] != 0) & (df[lbl_col] != 9999), lbl_col:lbl_col+4] # set custom column names x.columns = cols # set 'label' column as index (will be used for alignment by pd.concat()) x.index = x.label dfs.append(x) merge filtered DFs horizontally (aligned by indexes) result = pd.concat(dfs, axis=1) replace all label columns with the "most complete" list of labels result.ix[:, ::5] = pd.concat([result.index.to_series()] * (len(df.columns)//5), axis=1) replace all NaN's with 9999 result.fillna(9999, inplace=True) save resulting DF to Excel result.to_excel(out, index=False) out.xlsx: PS as it works http://pandas.pydata.org/pandas-docs/stable/indexing.html :In [27]: df Out[27]: a b c 0 4 9 8 1 8 4 5 2 7 9 4 3 4 7 1 4 0 7 3 5 8 1 4 6 1 9 8 7 4 9 3 8 1 3 8 9 7 9 0 to filter the lines in which a > 1 and b < 8:In [28]: df.ix[(df.a > 1) & (df.b < 8)] Out[28]: a b c 1 8 4 5 3 4 7 1 5 8 1 4 It's the same thing now, but it'sles to pick. ['a','b']:In [29]: df.ix[(df.a > 1) & (df.b < 8), ['a','b']] Out[29]: a b 1 8 4 3 4 7 5 8 1

Add one line in DataFrame easily using df.loc[len(df)] = [...]:In [9]: df Out[9]: ID Name First name Middle name Age Floor Weight Growth Family status 0 1 Name Yrij V 29 2 80 185 married In [10]: df.loc[len(df)] = [61, 'Василий', 'Флешков', 'Капустович', 512,'муж', 111, 121, 'х' ] In [11]: df Out[11]: ID Name First name Middle name Age Floor Weight Growth Family status 0 1 Name Yrij V 29 2 80 185 married 1 61 Василий Флешков Капустович 512 муж 111 121 х but it is much more efficient to build a set of lines and immediately create a ready DataFrame or create a new one and add it to the existing:In [12]: d = [[11, 'Bond', 'James', 'Jnr', 35, 'M', 85, 190, ''], ...: [12, 'Kong', 'King', '', 12, 'M', 2000, 600, ''] ...: ] ...: In [13]: new = pd.DataFrame(d, columns=df.columns) In [14]: new Out[14]: ID Name First name Middle name Age Floor Weight Growth Family status 0 11 Bond James Jnr 35 M 85 190 1 12 Kong King 12 M 2000 600 In [15]: df = df.append(new) In [16]: df Out[16]: ID Name First name Middle name Age Floor Weight Growth Family status 0 1 Name Yrij V 29 2 80 185 married 1 61 Василий Флешков Капустович 512 муж 111 121 х 0 11 Bond James Jnr 35 M 85 190 1 12 Kong King 12 M 2000 600 Well, actually, adding DataFrame to existing CSV file:df.to_csv(r'/path/to/file.csv', mode='a', header=None, index=False) PS I would not consider the CSV file as an OBD file as a maximum as a time storage facility (e.g. for transfer (export/import) to another OBD. Try MySQL / SQLite or HDF (the fast file repository is convenient for processing and retrieval of Pandas)

This is a working option that reads Excel the file only once:import pandas as pd fn = r'C:\Temp.data\primer.xlsx' cols = ['label','x','y','z','value'] df = pd.read_excel(fn, skiprows=4, header=None, parse_cols='C:XFD') dfs = [] for i in range(df.columns.size//5): lbl_col = 5*i x = df.ix[(df[lbl_col] != 0) & (df[lbl_col] != 9999), lbl_col:lbl_col+4] #x.columns = pd.MultiIndex.from_tuples(list(product([i+1], cols))) x.columns = cols dfs.append(x.reset_index(drop=True)) result = pd.concat(dfs, axis=1) #result.to_excel('c:/temp/result.xlsx', startrow=3, index=True) result.to_excel('c:/temp/result.xlsx', startrow=3, index=False) Result: PS I wanted to add the numbers of the five-string blocks (as in your original file: 1-19), i.e. to create multilevel columns: x.columns = pd.MultiIndex.from_tuples(list(product([i+1], cols))) but this has not yet been implemented:NotImplementedError: Writing to Excel with MultiIndex columns and no index ('index'=False) is not yet implemented. That is, it is necessary to select, either multi-level (red) names of columns and unnecessary indexes or no index and single-level names of columns. ♪ ♪PPS if you need a hat, you can do it. https://stackoverflow.com/a/38075046/5741205

Try using NumPy:import numpy as np a = np.asarray(f) usecols = np.argwhere(a==1).reshape(-1,) + 1 I don't know.In [30]: f = [1, 1, 0, 0, 1, 0, 1, 1] In [31]: a = np.asarray(f) In [32]: a Out[32]: array([1, 1, 0, 0, 1, 0, 1, 1]) In [33]: np.argwhere(a==1).reshape(-1,) Out[33]: array([0, 1, 4, 6, 7], dtype=int64) or following the logic of your program.if f[0]==1: s.append(1)] (i.e. with a change per unit):In [34]: np.argwhere(a==1).reshape(-1,) + 1 Out[34]: array([1, 2, 5, 7, 8], dtype=int64) PS usecols shall contain either the number of columns in the form of whole numbers (from scratch) or the names of the poles in the form of lines. ♪ ♪

Use the method. http://pandas.pydata.org/pandas-docs/stable/indexing.html#the-query-method-experimental :data = pd.read_csv('file.csv', sep=';', decimal=',', quotechar="'").query('seller != buyer') If you need to keep it back in CSV:data.to_csv('output.csv', index=False) PS you don't need to transpose DataFrame to filter it.PPS if you use Pandas, try not to use for loop - it's not very effective.Here's a working example. Considering that your CSV uses a quadrant quote ':CSV file - D:\temp\buyer_seller.csv:'id';'seller';'buyer';'timestamp' 1;seller-1;buyer-1;2016-01-01 2;seller-2;buyer-2;2016-01-02 3;same-1;same-1;2016-01-11 4;same-2;same-2;2016-01-22 Code:In [21]: pd.read_csv(r'D:\temp\buyer_seller.csv', sep=';') Out[21]: 'id' 'seller' 'buyer' 'timestamp' 0 1 seller-1 buyer-1 2016-01-01 1 2 seller-2 buyer-2 2016-01-02 2 3 same-1 same-1 2016-01-11 3 4 same-2 same-2 2016-01-22 In [22]: pd.read_csv(r'D:\temp\buyer_seller.csv', sep=';', quotechar="'") Out[22]: id seller buyer timestamp 0 1 seller-1 buyer-1 2016-01-01 1 2 seller-2 buyer-2 2016-01-02 2 3 same-1 same-1 2016-01-11 3 4 same-2 same-2 2016-01-22 In [23]: pd.read_csv(r'D:\temp\buyer_seller.csv', sep=';', quotechar="'").query('seller != buyer') Out[23]: id seller buyer timestamp 0 1 seller-1 buyer-1 2016-01-01 1 2 seller-2 buyer-2 2016-01-02 Alternatively, you can just get rid of the rubber in the names of the poles/colon:In [27]: df = pd.read_csv(r'D:\temp\buyer_seller.csv', sep=';') In [28]: df.columns.tolist() Out[28]: ["'id'", "'seller'", "'buyer'", "'timestamp'"] In [30]: df.columns = df.columns.str.replace("'", '') In [31]: df.columns.tolist() Out[31]: ['id', 'seller', 'buyer', 'timestamp']

UPDATE:D:\temp\price-bedrooms-addiction.png: This is a working example:import numpy as np import pandas as pd import matplotlib.pyplot as plt import matplotlib matplotlib.style.use('ggplot') т.к. вы не предоставили примера CSV воспользуемся случайными данными... df = pd.DataFrame(np.random.randint(0,100,(5, 20)),columns=['p{}'.format(i) for i in range(1,21)]) df['price'] = np.random.randint(105, 107, 5) for c in df.columns.difference(['price']).tolist(): df.set_index(c)['price'].plot.bar(stacked=True) plt.savefig('d:/temp/price-{}-addiction.png'.format(c)) Results:D:\temp\price-p1-addiction.png: D:\temp\price-p2-addiction.png: D:\temp\price-p3-addiction.png: etc.If you want to see price dependence on all parameters on the same schedule:In [98]: df.set_index('price').plot.bar(rot=0, stacked=True, figsize=(16, 12)) Out[98]: <matplotlib.axes._subplots.AxesSubplot at 0x932eb70>

Looks like you have a column. syear or indeed negative numbers or, most likely, large positive (more than 32767) which, when type of data np.int16 They turn into negative. ♪ ♪Demo:max() and min() for type np.int16:In [67]: np.iinfo(np.int16) Out[67]: iinfo(min=-32768, max=32767, dtype=int16) as a large positive whole (32880) is negative (32656) when using the type np.int16:In [72]: df = pd.DataFrame({'a':[32880]}, dtype=np.int16) In [73]: df Out[73]: a 0 -32656 Incorrect.In [88]: df1.query('syear <= 1980 or syear > 2016').syear Out[88]: 248737 -9076 248738 -26593 248739 1725 248740 -25171 248741 7963 248742 27137 248743 19854 248744 26738 248745 6716 248746 9885 248747 19361 248748 -19726 248749 -24605 248750 24074 248751 -8070 248752 -16027 248753 -23424 248754 3848 248755 1471 248756 30634 248757 -8162 248758 -18937 248759 16733 248760 -21923 248761 16817 248762 3834 248763 -13556 248764 -16229 248765 24272 248766 25642 ... 252510 -1 252511 -1 252512 -1 252513 -1 252514 -1 252515 -1 252516 -1 252517 -1 Name: syear, dtype: int16 Interesting surveillance - all bad data go on a continuous block (with indices: 248737 - 252517)

I don't think you're reading this file right.CParserError: Error tokenizing data. C error: Expected 1 fields in line 7, saw 10Says that Pandas saw 10 fields in line 7, and expected to see only one field. Try the following:fn = r'/path/to/file.csv' df = pd.read_csv(fn, skiprows=70, sep='\s*') PS I don't have access to your file, so I challenged my decision. https://www.researchgate.net/publication/269670937_GSE5147_series_matrixtxt A file in which the first 70 lines are described. Switch 70 to number of description lines in Your your yours. file

If I understood you correctly, I might need to. dataframe.merge()♪ Delete in one data frame a pole with zero, and then df1.merge(df2)♪

Number of cuts np.linspace(a, b, n) There must be a whole thing, so... n It's got to be a whole, not a shotgun, making it easier to do with a full-scale division. //:n = (b-a)//h + 1 In addition, the method .to_csv No parameter delimiterThere's a parameter. sep:df.to_csv('file.csv', index=False, sep=',') And commendable, of course, that you know how to combine the conditions in. PandasBut he understands. >=Two conditions don't need, one condition. In addition, .values No use when compared Series with number Pandas I understand that I'm being compared. valuesNot index:df = df[df.x >= b//2] P.S. The time-frame is fixed by the most elementary means itself. Pandas:df.plot(x='x', y='y');

I've got a pretty inconclusive design:import pandas as pd data = {'Parent Meta`s ID' : ['3516', '3518', '3517', '4440'], 'Child1' : [4451, 3607, 4143, 3598], 'Child2' : ["", "", 4451,3518], 'Child3' : ["", "","",3531]} df = pd.DataFrame(data) df["Child1"] = df.values.tolist() df = df.explode('Child1').iloc[:, :2] df["Level"] = 2-(df["Parent Metas ID"]==df["Child1"]) df.loc[df["Parent Metas ID"].duplicated(), "Parent Metas ID"]="" df.loc[df["Level"]==1, "Child1"]="" </code></pre><p>df:</p><pre><code> Parent Metas ID Child1 Level 0 3516 1 0 4451 2 0 2 0 2 1 3518 1 1 3607 2 1 2 1 2 2 3517 1 2 4143 2 2 4451 2 2 2 3 4440 1 3 3598 2 3 3518 2 3 3531 2 There is a suggestion that much can be simplified by compiling the data at the stage of their preparation, not after the date-freighting.

If I've understood the problem correctly, you can offer that solution. First, where there is no value in all columns NA We need to slightly modify the aggregating functions. By the way, you've got more than zero and less: "<0": (3, lambda x: x.lt(0).sum() if x.any() else np.NaN), "=0": (3, lambda x: x.eq(0).sum() if x.any() else np.NaN), ">0": (3, lambda x: x.gt(0).sum() if x.any() else np.NaN), And then need to be filled NA previous values:res1 = res1.ffill()

The answer of the algorithm algorithm is that the question has not been solved.create three additional columns with a number of minutes for each of the categories of packagingmultiply each of these columns to the appropriate tariff and override.

Try this:df["ID"] = df["ID"].fillna(df.groupby("Название")["ID"].transform("last")) result:In [273]: df Out[273]: Название ID Дата 0 Точка А 1.0 1.11 1 Точка А 2.0 2.11 2 Точка А 2.0 2.11 3 Точка B 4.0 2.11 4 Точка B 3.0 1.11 5 Точка B 4.0 2.11

Let's say the data set is called the array. Then the average number of columns Lam_F2_100, Lam_F2_2_101 and Lam_F2_2_102 in the mass ans is as follows:library(data.table) array <- as.data.table(array) array$Lam_F2_2_100 <- as.numeric(array$Lam_F2_2_100) array$Lam_F2_2_101 <- as.numeric(array$Lam_F2_2_101) array$Lam_F2_2_102 <- as.numeric(array$Lam_F2_2_102) ans <- array[, .(Lam_F2 = mean(sum(Lam_F2_2_100 , Lam_F2_2_101 , Lam_F2_2_102 )))] Similarly, the change in the columns within the sum( function) is the average of any column.

Something like that. If you only have these columns, you'll have an extra check. any I don't need it, especially since I clearly didn't write it properly.import pandas as pd df = pd.DataFrame({'int': [1, 2, 3], 'obj1': ['yes', 'no', 'yes'], 'obj2': ['no', 'yes', 'no'], 'obj3': ['n', 'y', 'n']}) obj_columns = df.columns[df.dtypes == 'object'].to_list() for col in obj_columns: if any(df[col].str.contains(x).any() for x in ('yes','no')): df[col] = df[col].map({'yes': 1, 'no': 0}) Out: int obj1 obj2 obj3 0 1 1 0 n 1 2 0 1 y 2 3 1 0 n

By mistake, in the friction. movies and ratings There's no single column with a fully matched name.The name of the columns is compared to the usual lines in Python. So if you don't have a match to the letters, or there are gaps in your name, the lines won't match.Check the name of the columns in both frictions:print(movies.columns.to_list()) print(ratings.columns.to_list()) and draw attention to the register of letters and gaps before or at the end of the columns

I need a column installs less than a day after day. serve_time, date in column installs replaced by date in column server_time.Well, literally, as it says, in those lines that satisfy the condition, change this for this:condition = df['installs'] < df['server_time'] df.loc[condition, 'installs'] = df.loc[condition, 'server_time']

Use the parameter. https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html :df = pd.read_csv("temp.csv", parse_dates=["Timestamp"]) result:In [114]: df.head() Out[114]: Timestamp F 0 2012-01-03 1 1 2012-01-06 1 2 2012-01-09 1 3 2012-01-12 1 4 2012-01-15 1 In [115]: df.dtypes Out[115]: Timestamp datetime64[ns] # <-- NOTE!!! F int64 dtype: object If there is at least one line in the data where time does not indicate midnight (00:00:00Then Pandas will show both date and time:In [116]: df = pd.read_csv("temp.csv", parse_dates=["Timestamp"]) In [117]: df.head() Out[117]: Timestamp F 0 2012-01-03 21:59:00 1 1 2012-01-06 00:00:00 1 2 2012-01-09 00:00:00 1 3 2012-01-12 00:00:00 1 4 2012-01-15 00:00:00 1 The time component can be cut as follows:In [118]: df["Timestamp"] = df["Timestamp"].dt.floor("D") In [119]: df.head() Out[119]: Timestamp F 0 2012-01-03 1 1 2012-01-06 1 2 2012-01-09 1 3 2012-01-12 1 4 2012-01-15 1 If you need to convert the date to a given format, it's done:df["timestamp_str"] = df["Timestamp"].dt.strftime("%m/%d/%Y") result:In [121]: df.head() Out[121]: Timestamp F timestamp_str 0 2012-01-03 1 01/03/2012 1 2012-01-06 1 01/06/2012 2 2012-01-09 1 01/09/2012 3 2012-01-12 1 01/12/2012 4 2012-01-15 1 01/15/2012 In [122]: df.dtypes Out[122]: Timestamp datetime64[ns] F int64 timestamp_str object # <-- NOTE!!! dtype: object NOTE: After this transformation, you will get the lines instead of the dates and, as a consequence, you will not be able to use any function to deal with dates/time, including arithmetic dates. Usually, dates are converted to a specific format only before they are retained in text form or for display/graph.

What prevents the introduction of a new index and simple content?df1 = df1.set_index("name") df2 = df2.set_index("name") res = pd.concat([df1, df2]) res: id count name 48/a a 2 48/c c 5 50/a a 5 50/b b 6 50/d d 6 Necessary columns can be removed and, if there is a wish, reset_index(s)UPDATEIf the number of columns may vary, but the columns you need all have the same name and meaning in the column. name It's unique, so we can try this:res1 = pd.merge(res1[['name', 'count']], df2[['name', 'count']], how="outer") res1: name count 0 48/a 2 1 48/c 5 2 50/a 5 3 50/b 6 4 50/d 6