In your frame date, use the .max() in the product column, I think this can help you if it's really what I understood the question.I can't see the image.

What you are looking for is called "stopwords" and is a kind of filtering traditionally used in natural language processing. Here's an example using the nltk package:import nltk nltk.download('stopwords') from nltk.corpus import stopwords sw = stopwords.words('portuguese') text= ''' Processamento de língua natural (PLN) é uma subárea da ciência da computação, inteligência artificial e da linguística que estuda os problemas da geração e compreensão automática de línguas humanas naturais. Sistemas de geração de língua natural convertem informação de bancos de dados de computadores em linguagem compreensível ao ser humano e sistemas de compreensão de língua natural convertem ocorrências de linguagem humana em representações mais formais, mais facilmente manipuláveis por programas de computador. Alguns desafios do PLN são compreensão de língua natural, fazer com que computadores extraiam sentido de linguagem humana ou natural e geração de língua natural. ''' new_text = ' '.join([k for k in text.split(" ") if k not in sw]) print(new_text) Return:Processamento língua natural (PLN) subárea ciência computação, inteligência artificial linguística estuda problemas geração compreensão automática línguas humanas naturais. Sistemas geração língua natural convertem informação bancos dados computadores linguagem compreensível ser humano sistemas compreensão língua natural convertem ocorrências linguagem humana representações formais, facilmente manipuláveis programas computador. Alguns desafios PLN compreensão língua natural, fazer computadores extraiam sentido linguagem humana natural geração língua natural. You could do the filtration by manually setting the prepositions, as suggested in the comments, but the chance of error is higher.

I believe groupby with agg be the solution. Here’s the example:Loading librariesimport pandas as pd import random Auxiliary variablessexos = ["M", "F"] ufs = ["BA", "ES", "MT", "RJ", "SP"] cores = ["amarelo", "azul", "vermelho", "roxo", "verde"] tamanho = 1000 Creating test dataframedf = pd.DataFrame({"sexo": [random.choice(sexos) for _ in range(tamanho)], "uf": [random.choice(ufs) for _ in range(tamanho)], "cor": [random.choice(cores) for _ in range(tamanho)]}) Creating "pivot_table"print(df.groupby(["sexo", "uf", "cor"]).agg({"cor": 'count'})) The result will be something like below: cor sexo uf cor F BA amarelo 21 azul 20 roxo 14 verde 15 vermelho 25 ES amarelo 23 azul 23 roxo 22 verde 24 vermelho 25 MT amarelo 24 azul 19 roxo 16 verde 26 vermelho 14 RJ amarelo 14 azul 20 roxo 14 verde 21 vermelho 14 SP amarelo 20 azul 24 roxo 17 verde 18 vermelho 14 M BA amarelo 15 azul 17 roxo 24 verde 22 vermelho 30 ES amarelo 14 azul 19 roxo 18 verde 22 vermelho 20 MT amarelo 33 azul 19 roxo 15 verde 18 vermelho 22 RJ amarelo 29 azul 18 roxo 20 verde 19 vermelho 19 SP amarelo 17 azul 22 roxo 17 verde 22 vermelho 22

We have two scenarios:Columns that have integers that should be stringColumns that have string that should be wholeFor the first case, for example, the whole case 1 have to turn into string 1, the solution is simple: just use the astypeExampledf["coluna"] = df["coluna"].astype(str) In the second case we have two possibilities:a. All values that have to be converted from string to int (float) can be convertedb. Some (or multiple, or all) values that have to be converted from string to int (float) they can't be convertedIn case all values can be converted, just use the same solution already described:df["coluna"] = df["coluna"].astype(int) For the second case, see the example:Create a Transform function to int or return nanimport numpy as np def to_int(row): try: return int(row) except ValueError: return np.nan Using the function in a dataframedf = pd.DataFrame({"A": [1, "a", 3, "4"]}) print(df) A 0 1 1 a 2 3 3 4 df["A"] = df["A"].apply(to_int) print(df) A 0 1.0 1 NaN 2 3.0 3 4.0 EDITTED on 10/08/2021 - motif: comment belowIf the 10 is whole, do:df["A"].apply(lambda x: x if isinstance(x, str) else np.nan) If 10 is string type, do something like:def only_strings(row): try: int(row) return np.nan except ValueError: return row and call withdf["A"] = df["A"].apply(only_string) End of edition

First create a column with the age group (may be using pd.cut as you tried), then group for it:df = ... # seu DataFrame aqui faixas_etarias = np.arange(0, df['idade'].max()+1, 20) df['faixa_etaria'] = pd.cut(df['idade'], faixas_etarias) A partir daqui, use df.groupby para agrupar pela coluna 'faixa_etaria' df_agrupado = df.groupby('faixa_etaria') Note that df_agrupado is a type object DataFrameGroupBy, on which you will normally want to do an operation that will be applied to each group. For example, df_agrupado['idade'].mean() will show the average age within each age group.

You can take any method / attribute of an object through a string with the function built-in getattr:import numpy as np a = np.array([1, 2, 3]) a linha abaixo equivale a escrever "soma = a.sum()" soma = getattr(a, 'sum')() print(soma) # output = 6 Without example data, I can't test if your code logic is working. But focusing on your question, you can do the operation with:df_join[feat + '_' + medida] = getattr(df_join[feat], medida)()

To verify that there are letters you can use https://pandas.pydata.org/docs/reference/api/pandas.Series.str.contains.html#pandas.Series.str.contains passing as a parameter for the search a RegExp as r'[a-zA-Z]' to create a true table where letters are found.And to check the amount of dimensions you can count the occurrence of spaces https://pandas.pydata.org/docs/reference/api/pandas.Series.str.count.html#pandas-series-str-count Ex:import pandas as pd dados = { "Código": [1623,5124,4785,5251,1578,9684,5187, 111, 222, 333, 444, 555 ], "Dimensões":["10 15 7", "15 7 8", "20 15 8", "50 70 22", "c10 l20", "50 10 2", "75 20 10", "10 cl20", "1c0 l20", "111 212", "111 22 33", "11 22 33 44"], } Criar o DataFrame df = pd.DataFrame( dados ) Criar uma TABELA VERDADE para selecionar as linhas que contem letras na coluna 'Dimensões' com_letras = df['Dimensões'].str.contains(r'[A-Za-z]') print( "# tabela verdade de linhas que contem letras na coluna 'Dimensões':" ) print( com_letras ) print( "\n") print( "# df de linhas COM letras na coluna 'Dimensões':") print( df[ com_letras ] ) print( "\n") Criar uma TABELA VERDADE para selecionar as linhas não tem 2 espaços na coluna 'Dimensões': dif_2espacos = df['Dimensões'].str.count(' ') != 2 print( "# tabela verdade de linhas não tem 2 espaços na coluna 'Dimensões':") print(dif_2espacos) print( "\n") print( "# df de linhas não tem 2 espaços na coluna 'Dimensões':") print(df[ dif_2espacos ]) print( "\n") Agora você pode utilizar bitwise ~ (NOT) para negar uma TABELA VERDADE e dessa forma transformar quem é TRUE em FALSE e vice-versa e depois bitwise & (AND) onde ambas tabelas sejam verdadeiras, ou seja: sem_letras = ~com_letras tem_2espacos = ~dif_2espacos sem_letras_E_tem_2espacos = sem_letras & tem_2espacos df = df[ sem_letras_E_tem_2espacos ].copy() df = df[ ~tem_letras & ~dif_2espacos ].copy() print("# df de linhas ( SEM letras AND COM 2 espacos ) na coluna 'Dimensões' :") print(df) https://ideone.com/QqYZL9 TL;DR;df[ ~df['Dimensões'].str.contains(r'[A-Za-z]') & (df['Dimensões'].str.count(' ') == 2) ] https://ideone.com/yIyUNy

I suggest you download an application that aims to split a giant CSV file into small pieces, that is, in several smaller CSV files that will present a number of lines defined by you.I suggest the app It's a simple application, just run, no need to install, I've used it several times. If you want to see other application options, see this website https://www.makeuseof.com/tag/how-to-split-a-huge-csv-excel-workbook-into-seperate-files/ Private CaseTo solve your particular case, I downloaded your file and split it into files with 100,000 lines. As you said you’d like to work with 500 thousand lines or less, I’m offering a zip file with 5 CSVs of 100,000 lines, just click https://drive.google.com/file/d/1es3Z5W3D7oyqmquGHhsJ6h3QHW0sDdZS/view?usp=sharing

The problem should be in the way the data is read from Google Drive. The link file is a comma CSV file as a marker for decimals. And the column separator is the semicolon, ";". To read files in this CSV format you must use read.csv2.library(gamlss) ModelGamlss <- gamlss( log(Var1) ~ re(fixed = ~ log(Var2) + log(Var3) + Var4 + Var5, random = list(Var6 = pdDiag(~ Var7))), data = Data2) #GAMLSS-RS iteration 1: Global Deviance = -1706.294 #GAMLSS-RS iteration 2: Global Deviance = -1706.294 summary(ModelGamlss) #****************************************************************** #Family: c("NO", "Normal") #Call: gamlss(formula = log(Var1) ~ re(fixed = ~log(Var2) + log(Var3) + Var4 + Var5, random = list(Var6 = pdDiag(~Var7))), data = Data2) #Fitting method: RS() #------------------------------------------------------------------ #Mu link function: identity #Mu Coefficients: Estimate Std. Error t value Pr(>|t|) #(Intercept) 3.100661 0.002601 1192 <2e-16 *** #--- #Signif. codes: 0 ‘’ 0.001 ‘’ 0.01 ‘’ 0.05 ‘.’ 0.1 ‘ ’ 1 #------------------------------------------------------------------ #Sigma link function: log #Sigma Coefficients: Estimate Std. Error t value Pr(>|t|) #(Intercept) -2.70187 0.02742 -98.53 <2e-16 *** #--- #Signif. codes: 0 ‘’ 0.001 ‘’ 0.01 ‘’ 0.05 ‘.’ 0.1 ‘ ’ 1 #------------------------------------------------------------------ #NOTE: Additive smoothing terms exist in the formulas: i) Std. Error for smoothers are for the linear effect only. #ii) Std. Error for the linear terms maybe are not accurate. #------------------------------------------------------------------ #No. of observations in the fit: 665 #Degrees of Freedom for the fit: 49.27846 Residual Deg. of Freedom: 615.7215 at cycle: 2 #Global Deviance: -1706.294 AIC: -1607.737 SBC: -1385.995 #****************************************************************** Data readingFrom help("read.csv2"), my emphasis:read.csv and read.csv2 are identical to read. table except for the defaults. They are intended for reading ‘comma separated value’ files (‘.csv’) or (read.csv2) the variant used in countries that use a comma as decimal point and a semicolon as field separator.Translation Google, edited:read.csv and read.csv2 are identical to read. table, except for patterns. They are intended for reading files of ‘comma separated values’ (‘.csv’) or (read.csv2) the variant used in countries that use a comma as a decimal point and a comma point as a field separator.google_id <- "1q4XuSPnZXx7TsrT3JVoucFyxeOSegWFz" google_file <- sprintf("https://docs.google.com/uc?id=%s&export=download", google_id) Data2 <- read.csv2(google_file) str(Data2) #'data.frame': 665 obs. of 7 variables: $ Var6: chr "A5A" "A5A" "A5A" "A5A" ... $ Var2: num 15.8 15.8 15.8 15.8 15.8 15.8 15.8 15.8 15.8 15.8 ... $ Var1: num 15.1 10.9 13.2 13.5 16.8 15.4 16.1 15.2 15.3 13.9 ... $ Var3: num 0.817 1.512 1.267 1.262 0.754 ... $ Var4: num 0.0364 0.0674 0.0564 0.0562 0.0336 ... $ Var5: num 0.0801 0.1482 0.1242 0.1237 0.0739 ... $ Var7: int 0 0 0 0 0 0 0 0 0 1 ...

It has a way to do what you want, but it is not as straight as you would like:Create a dedicated function to implement the calculation of the bmi according to your criteria.Separately create column values b) with function for all lines.Use the method where() to replace the values Na by their calculated values.It's an overkill, you have to calculate this replacement value for all lines instead of just the ones you need, but it works.# função que retorna o índice bmi para um valor de idade def median(df, age): if age >= 18 and age < 30: return df.loc[(df.age >= 18) & (df.age < 30)].bmi.mean() elif age >= 30 and age < 45: return df.loc[(df.age >= 30) & (df.age < 45)].bmi.mean() elif age >= 45 and age < 65: return df.loc[(df.age >= 45) & (df.age < 65)].bmi.mean() elif age >= 65: return df.loc[df.age >= 65].bmi.mean() lista com o valor bmi para todas as entradas na coluna age new_bmi = [median(base, age) for age in base.age] data frame com a coluna bmi substituída pelos valores novos somente onde tem NaN base.bmi = base.bmi.where(base.bmi.notna(), new_bmi) Take the test there with your frame date and let me know if you've solved it. Hug.

It would be cool you post the date frame but you could do something like this: df.groupby(['|ID_Cliente','Date_Criação'], as_index=False)['Compra'].sum().sort_values(by='Compra', ascending=False)Supposing that Data_Criaçãobe the field with duplicates and Compra be the field you want to add.

Whereas the condition described below is true:For each row, columns have either the same value or NaNSo it is enough to make the average of them, because the method mean ignores NaNExample>>> import pandas as pd >>> df = pd.DataFrame({"col1": [10, 20, 30], "col2": [None,20,30], "col3": [None,None,30]}) >>> df col1 col2 col3 0 10 NaN NaN 1 20 20.0 NaN 2 30 30.0 30.0 >>> df['media'] = df.mean(axis=1) >>> df col1 col2 col3 media 0 10 NaN NaN 10.0 1 20 20.0 NaN 20.0 2 30 30.0 30.0 30.0 I hope it helps

Have you tried using Scikit-Learn to perform normalization? For this, you can use the class StandardScaler that performs this task as follows:from sklearn.preprocessing import StandardScaler norm = StandardScaler() d1_norm = norm.fit_transform(d1) d2_norm = norm.fit_transform(d2)

It can use filter strategies to avoid peaks and points out of the curve, in various data domains (that is - data from various areas of knowledge: in health an out value could identify a wrong copied exam measure, or taken at a time of stress, for example). But s is using for asset quotes - I believe it makes sense to use the values as they are - taking out the "outliers" would end any usefulness of your model - If the closing value on the 2/12 day was 3% higher, it was 3% higher - it is not a wrong configuration in a photo, an electrical noise in an analog instrument, etc... you have to take this variation into account.Moreover, it may be that precisely because of this feature of market data - (they are already imputed from digital systems), you simply do not have none point that would be an "outlier" in the data you're working on - may even be that you've done everything right.As you did not put a way for us to help you more concretely: neither the code you are using to plot the data, nor a way you are responding to having the dataframe to create some example plots, it is not possible to help you beyond that point.It's easy to find articles on the subject, but I suppose you've already been through some. This one seems reasonably easy to follow and complete: https://towardsdatascience.com/ways-to-detect-and-remove-the-outliers-404d16608dba

You can solve this by using the function .map() passing a Series to her, like this:dataset_original = pd.DataFrame({'Grau de instrucao': ['1','2','3','4','5','6','7','8','9','10','11','-1']}) s = Grau_Instr_Bibli.set_index('Valores na fonte')['Categoria'] dataset_original['GrauNovo'] = dataset_original['Grau de instrucao'].map(s) #saida do dataset_original Grau de instrucao GrauNovo 0 1 Analfabeto 1 2 Até 5ª Incompleto 2 3 5ª Completo Fundamental 3 4 6ª a 9ª Fundamental 4 5 Fundamental Completo 5 6 Médio Incompleto 6 7 Médio Completo 7 8 Superior Incompleto 8 9 Superior Completo 9 10 MESTRADO 10 11 DOUTORADO 11 -1 IGNORADO

Your code has two problems: First your problem is happening because your day_mes_year variable will only store the last value of the loop, this makes all your months equal. According to the pandas is an eficent tool to work very large worksheets. If you make a loop that goes through the entire worksheet you will do it less efficiently than the pandas is able to do. See a much more efficient way to use pandas features to get the months without having to sweep the worksheet:dfdados = pd.read_csv('DadosClimaticos2018Londrina.csv', sep =';') meses = dfdados.Data.str.split('/').str[1] dfdados['Mês']=meses realize that the command meses = dfdados.Data.str.split('/').str[1] does exactly the same thing you wanted: a split of the month and the selection of the element in position 1 of the list. The difference is that it does this on all elements of the Data column at once

I believe you should carry out the following command in your terminal.pip install pandas Remember that after doing this pip procedure, you should put it at the beginning of your code:import pandas as pd Then the pandas module will be executed successfully!It should be remembered that if you are trying to do this locally it is recommended that you install the environment variable. Follows a link to how to perform this procedure: https://medium.com/@victorromariopazdejesus/python-3-configurando-vari%C3%A1veis-de-ambiente-no-windows-10-63059c7192e6

... # get_all_values gives a list of rows. rows = worksheet.get_all_values() print(rows) ... Import of pandasimport pandas as pd One output is to take the first row and turn into columns:df = pd.DataFrame.from_records(rows, columns = rows[0] ) And then remove the first line:df.drop(0, axis = 0, inplace = True) Another possible output is to manually place the column names.

Imports:import pandas as pd import matplotlib.pyplot as plt Loading date frame:df_escolaridade = pd.read_csv('./dataframe.csv', index_col=[0]) df_escolaridade.drop(columns = 'Código', inplace = True) Checking the data types:df_escolaridade.dtypes % dos ocupados com fundamental completo - 18 anos ou mais 2010 float64 % dos ocupados com médio completo - 18 anos ou mais 2010 float64 % dos ocupados com superior completo - 18 anos ou mais 2010 float64 % de 18 a 20 anos com médio completo 2010 float64 % de 15 a 17 anos com fundamental completo 2010 float64 % de 18 a 24 anos com fundamental completo 2010 float64 % de 18 anos ou mais com fundamental completo 2010 float64 % de 25 anos ou mais com fundamental completo 2010 float64 % de 18 a 24 anos com médio completo 2010 float64 % de 18 anos ou mais com médio completo 2010 float64 % de 25 anos ou mais com médio completo 2010 float64 % de 25 anos ou mais com superior completo 2010 float64 Expectativa de anos de estudo 2010 float64 This is where we move the columns to the lines, making the pivot.'Pivoting' the date frame:df_pivotado = pd.pivot_table(df_escolaridade, values = df_escolaridade.columns, columns='Espacialidades') Plotting the chart:df_pivotado.plot(kind = 'barh', figsize = (15,10))

Imports:import pandas as pd Loading data:df = pd.read_csv('https://raw.githubusercontent.com/Renatolopo62/IFNMG-Januaria/master/dados/dados_padronizados_matriculas_januaria_2009_2018.csv') Converting the entire frame date to string type to be able to manipulate:df1 = df.applymap(str) Selecting which columns will generate the new column ID:df1['ID'] = df1[['TP_COR_RACA', 'TP_SEXO', 'NU_ANO_NASCIMENTO', 'NU_MES_NASCIMENTO', 'NU_DIA_NASCIMENTO']].agg(''.join, axis=1) Displaying the ID columndf1['ID'] Output:0 311990613 1 60198987 2 601990130 3 611990119 4 101981816 ... 10982 321999117 10983 11199996 10984 321991615 10985 312000426 10986 321998121