There is a Moodle table, "mdl_log", which saves all the information from the user activity on the platform.

In case it's good, I've already found the solution. You can read it here: https://handbook-5-1.cochrane.org/chapter_16/16_1_3_2_imputing_standard_deviations_for_changes_from_baseline.htm

Function estadisticas you have to call with a x and one and are lists with numerical values.A serious example:import pandas as pd import numpy as np columnas = ['nombre', 'Matematicas', 'Ciencias', 'Espanol', 'Historia', 'EdFisica'] lineas = [['Lucia ', 7.0, 6.5, 9.2, 8.6, 8.0], ['Pedro ', 7.5, 9.4, 7.3, 7.0, 7.0], ['Ines ', 7.6, 9.2, 8.0, 8.0, 7.5], ['Luis ', 5.0, 6.5, 6.5, 7.0, 9.0], ['Andres', 6.0, 6.0, 7.8, 8.9, 7.3], ['Ana ', 7.8, 9.6, 7.7, 8.0, 6.5], ['Carlos', 6.3, 6.4, 8.2, 9.0, 7.2], ['Jose ', 7.9, 9.7, 7.5, 8.0, 6.0], ['Sonia ', 6.0, 6.0, 6.5, 5.5, 8.7], ['Maria ', 6.8, 7.2, 8.7, 9.0, 7.0]] df = pd.DataFrame(columns=columnas, data=lineas) df.set_index('nombre', inplace=True) y = df['Espanol'] x = df['Matematicas'] def estadisticas(x, y): return {'Variable_1': x, 'Variable_2': y, 'Correlacion': np.corrcoef(x, y), 'Covarianza': np.cov(x, y)} estadisticas(x, y) To work with the dataframe, you can write the function with a dataframe and the names of two columns. And to compare every 2 columns, https://docs.python.org/3/library/itertools.html is a comfortable way to catch each combination:import itertools def estadisticas(df, col_x, col_y): return {'Variable_1': col_x, 'Variable_2': col_y, 'Correlacion': np.corrcoef(df[col_x], df[col_y]), 'Covarianza': np.cov(df[col_x], df[col_y])} for col_x, col_y in itertools.combinations(df.columns, 2): print(estadisticas(df, col_x, col_y )) The most comfortable thing to visualize is the function https://seaborn.pydata.org/generated/seaborn.pairplot.html of the bookstore seaborn:from matplotlib import pyplot as plt import seaborn as sns df = .... g = sns.pairplot(df) plt.tight_layout() plt.show() You'd get:

It is only enough to create an array in which you will store the values and to access them you can do it through subscripts, for example:numero_de_datos = int(input("introduzca el numero de datos: ")) #Lista donde se almacenarán los datos. datos = [] #Dandole valores a la lista vacía datos. for i in range(numero_de_datos): datos.append(float(input("introduzca el valor{}: ".format(i+1)))) Afterwards, to obtain the data from the list, simply use subscripts:for j in range(numero_de_datos): print(datos[j])

Your problem is to understand that they serve the phases of training, Validation and test. First I give you an excerpt from Brian Ripley, a well-known statesman from Oxford University who brought great advances to the world of data science:- Training set: A set of examples used for learning, that is to fit the parameters of the classifier.- Validation set: A set of examples used to tune the parameters of a classifier, for example to choose the number of hidden units in a neural network.- Test set: A set of examples used only to assess the performance of a fully-specified classifier.Brian Ripley, page 354, Pattern Recognition and Neural Networks, 1996Now I go to explain it in Spanish and with my own words:Training (Train)The set of training data is the one used to calculate the coefficients of some algorithm. The coefficients are the Betas (β), the simplest case of seeing is a linear regression. Here the algorithm will calculate the weights of each β, to apply it to each variable "X_n"This dataset, is the one shown to the algorithm and uses to learnValidation (Validation)This set is used to calculate the best hyperparameters in your model. Once your model has been trained with hyperparameters, the validation set is used, which "there's data the model has never seen before." and with this set different hyperparameters are tested to see what works best.For example, in your algorithm MLPRegressor You put a number of hidden layers (30, 30)But maybe (50, 50) work better with your dataset, or (60, 60). Or maybe the activation leakyrelu It's better than that. relu.TestOnce the algorithm training has been done, and it has been proven that hyperparameters work better, in a set of "not previously seen" data, the set of testThese data are data that you have never seen the model, and are used to see how the trained model would work, with the hyperparameters selected in the real world.Conclusions and bonusAs the model's hyperameters are adjusted using the validation set, the normal thing is that the error is always greater in the test set than in the validation setYou must never skip these three steps Train - rigid Validation - voluntary test. An algorithm that works well in validation, can do it lousy in test.There are methods that show the validation set to the model at the time of training, and have been proven very effective, as https://en.wikipedia.org/wiki/Cross-validation_(statistics) , https://en.wikipedia.org/wiki/Cross-validation_(statistics)#k-fold_cross-validation etc. (That's why I put on quotation marks "there's data the model has never seen before.")The ratios to be used when dividing your dataset are usually between 50% - 60% for training, and 20% - 25% for validation and test. Although this will depend on the amount of data you haveNever under any concept, test data is used to train the model, as @Christian indicates in the comments would be how to copy into a test. The aim of the test set is to see how the model would work in a real production environmentin which the data will not be knownValidation and test with Sklearn are performed the same, once the model has been trained, is used modelo.predict(X_val) to predict results, and then apply some metric, such as accuracy to check how good the predictions are.

When the structure comes in a dictionary, the easiest thing is to pass it to Pandas since dictionaries is the structure you use Pandas to create their DataFrames. You can do the following:import pandas as pd df_list = [] for participant in participants: df = pd.DataFrame(participant["joueurs"]) df["date"] = participant["date"] df_list.append(df) df_jugadores = pd.concat(df_list) Departure: name sex date 0 Pierre-Irénée Perrin m 2-6-2021 1 Ivanne Combe Laboissière f 2-6-2021 2 Domitille Croizier des Hormais f 2-6-2021 3 Ombeline d'Harcourt f 2-6-2021 4 Blanche d'Harcourt f 2-6-2021 ... Once you have it in structure DataFrame You can already operate the results more easily, without having to operate and create functions for dictionaries, which is more tedious.

You could use the module pandas to make the calculation easily using dataframes:To install it you can use pip3pip3 install pandas If you are in a distribution like ubuntu or debian you can install pip:apt-get update && apt-get install python3-pip Depending on the distribution you have, you can already use the command.As for the code:# Importas el módulo: import pandas as pd Cargas el contenido del .csv en el dataframe df = pd.read_csv('fruits.csv') Agrupas por la columna 'Fruit' y haces la media por 'Value' result = df.groupby('Fruit')[['Value']].mean() Si imprimimos resultado: print(result) We get:$-> python3 average.py Value Fruit Arándano 2.666667 Cereza 3.500000 Manzana 1.600000

You can do something like that:import pandas as pd pd.Series(df[["uno", "dos", "tres"]].values.flatten()).mode()[0] With df.values We get a array of the original and https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.ndarray.flatten.html#numpy-ndarray-flatten "we flatten" it into one of a single dimension. The only thing that would be left is to use pd.Series() to transform it into a Serie and can use mode()

With numpy there is no function that fits fashion. You can use the statistics package that is already installed on python, or use scipy.Using statistics would be like this:import statistics as stat edades = [21, 17, 89, 76, 32, 21, 45, 21, 89, 21, 15, 89, 21] moda=stat.mode(edades) print(moda) In this case the fashion is 21.And using scipy would be like this:from scipy import stats as st edades = [21, 17, 89, 76, 32, 21, 45, 21, 89, 21, 15, 89, 21] moda=st.mode(edades) print(moda) The result is: ModeResult(mode=array([21]), count=array([5]))Which means the most repeated value is 21 with a total of 5 times.Greetings!

One way to achieve it would be by extracting the chart data. To do this it is enough to pass the parameter return.grid=TRUEto function plot.gbmlibrary(gbm) data(iris) str(iris) GBM.model = gbm(Species ~ Sepal.Length + Sepal.Width + Petal.Length + Petal.Width, data = iris,distribution = "multinomial") #Guardamos los datos en X X<-plot.gbm(GBM.model,"Petal.Length",return.grid=TRUE) Once we have the data stored in X, we can transform them to gratify them for example ggplot2 or some other graphic bookstore. In this case the package gbm Use lattice for your graphics and we could make a graphic equal to your example with the following code#Pasamos los datos a formato lagro df<-NULL for(i in colnames(X$y)){ #Atencion #X es un data.frame cuya columna "y" es una matriz #Para pasarla a formato ancho #Cada una de las columnas de la matriz "y" se pasan a la columna y del nuevo df #En la columna Petal.Length de df siempre se pasa la misma columna de X #Y en la columna Species (Por la que se agrupa para dar color) #se pasa el nombre de la columna de la matriz "y" aux&lt;-data.frame(y=X$y[,i],Petal.Length=X$Petal.Length,Species=i) df&lt;-rbind(df,aux) } #Mostramos un resumen de los datos str(df) #Gráfica usando el paquete lattice lattice::xyplot(y ~ Petal.Length, data = df, type = "l", groups=Species, ylab="f(Petal.Length)")

This can be addressed in two additional ways: Enable Asynchronous Auto Update Statistics and the database level (not my preference). If your production is stuck because due to this issue then this is the quick fix as it is instantaneous but it globally affects the database and can produce sub-par query plans since the plan is generated prior to the stats update. ALTER DATABASE [DatabaseName] SET AUTO_UPDATE_STATISTICS_ASYNC ON WITH NO_WAIT Remove the auto generate stat and replace it with a manually created one with no-recompute. The downside is that, while this is a surgical approach, it takes some time to create the statistic and will cause blocking. DROP STATISTICS [dbo].[TableName].[_WA_Sys_00000004_7D78A4E7] GO CREATE STATISTICS [_Manual_Stat_Name] ON [dbo].[TableName]([Column]) WITH NORECOMPUTE; GO When we were forced to deal with a similar issue we went with option 1 first and later option 2. During a "green zone" we reverted the database wide setting and then replaced the column's statistic with the manually created NORECOMPUTE one.

Indexes store actual data (data pages or index pages depending on the type of index we are talking about), and Statistics store data distribution. Therefore, CREATE INDEX will be the DDL to create an index (clustered, nonclustered, etc.) and CREATE STATISTICS is the DDL to create the statistics on columns within the table. I recommend you read about these aspects of relational data. Below are a couple of beginner, introductory articles. These are very broad topics, and therefore the information on them can go very wide and very deep. Read up on the general idea of them below, and ask more specific questions when they arise. BOL reference on Table and Index Organization BOL reference on Clustered Index Structure BOL reference on Nonclustered Index Structures SQL Server Central on the Introduction to Indexes BOL reference on Statistics Here is a working example to see these two parts in action (commented to explain): use testdb; go create table MyTable1 ( id int identity(1, 1) not null, my_int_col int not null ); go insert into MyTable1(my_int_col) values(1); go 100 -- this statement will create a clustered index -- on MyTable1. The index key is the id field -- but due to the nature of a clustered index -- it will contain all of the table data create clustered index MyTable1_CI on MyTable1(id); go -- by default, SQL Server will create a statistics -- on this index. Here is proof. We see a stat created -- with the name of the index, and the consisting stat -- column of the index key column select s.name as stats_name, c.name as column_name from sys.stats s inner join sys.stats_columns sc on s.object_id = sc.object_id and s.stats_id = sc.stats_id inner join sys.columns c on sc.object_id = c.object_id and sc.column_id = c.column_id where s.object_id = object_id('MyTable1'); -- here is a standalone statistics on a single column create statistics MyTable1_MyIntCol on MyTable1(my_int_col); go -- now look at the statistics that exist on the table. -- we have the additional statistics that's not necessarily -- corresponding to an index select s.name as stats_name, c.name as column_name from sys.stats s inner join sys.stats_columns sc on s.object_id = sc.object_id and s.stats_id = sc.stats_id inner join sys.columns c on sc.object_id = c.object_id and sc.column_id = c.column_id where s.object_id = object_id('MyTable1'); -- what is a stat look like? run DBCC SHOW_STATISTICS -- to get a better idea of what is stored dbcc show_statistics('MyTable1', 'MyTable1_CI'); go Here is what a test sample of statistics can look like: Notice that Statistics are the containment of the data distribution. They help SQL Server determine an optimal plan. A good example of this is, imagine you are going to life a heavy object. If you knew how much that weight because there was a weight marking on it, you'd determine the best way to lift and with what muscles. That's sort of what SQL Server does with statistics. -- create a nonclustered index -- with the key column as my_int_col create index IX_MyTable1_MyIntCol on MyTable1(my_int_col); go -- let's look at this index select object_name(object_id) as object_name, name as index_name, index_id, type_desc, is_unique, fill_factor from sys.indexes where name = 'IX_MyTable1_MyIntCol'; -- now let's see some physical aspects -- of this particular index -- (I retrieved index_id from the above query) select * from sys.dm_db_index_physical_stats ( db_id('TestDB'), object_id('MyTable1'), 4, null, 'detailed' ); We can see from the example above that the index actually contains data (depending on the type of index, leaf pages will be different). This post has only shown a very very very brief overview of these two large aspects SQL Server. Both of these could take up chapters, and books. Read some of the references, and then you will have a better grasp.

A number of factors can change on import. Fill factor for data blocks is likely more optimal. Partially filled data blocks from the development system will be filled on import. This can change over time if data is updated and added, but does not appear to be a factor in this case. Changes in block placement of rows may alter the number of block required to retrieve by index. This can have positive or negative impacts on the data. Cost parameter for data and or index blocks may be different on the two databases resulting in different cost weights for plans. Block sizes may be different on the two databases resulting in different costs. Newer versions of Oracle will recalculate statistics automatically as data changes. The exported statistics are normally a subset of the full statistics. There is a DBA_STATS package which can be used to export and import the full set of statistics. Histogram sizes may be different on the two systems resulting in differences in level of detail. (Its been a while since I tuned this kind of thing.) If the histograms are missing, Oracle will sample the data and estimate. Different sample sizes can cause very different behavior. Oracle may decide to use bitmap indexes in some cases. This can require regenerating the indexes on each query. You can add bitmap indexes for the required columns. This option can be disabled at the database level (it may be an undocumented setting). Try generating cost statistics for the queries you are using on both the Development and Production systems. You may also want to examine the Statistics on both systems. You may get more consistent results using transportable tablespaces for the tables and indexes. This will eliminate the import/export step. EDIT: Added bitmap, statistics, and histogram info above. It is possible to just run an explain plan without running the queries. There are various tools which assist in this. I have used Aqua Data Studio, but SQL Developer should work as well. For parsed queries (running or not) there is a view available from which the plan can be extracted. There are available views which can be used to view the current statistics for a table. Comparing used blocks on both systems will give you a quick idea how much statistics should have changed. Histogram data is can result in different plans.

You are running RTM (10.50.1600.1): the latest patch is SP2 + CU3 (10.50.4266) These include patches like KB 2498786

Your select on dba_tables doesn't take into account: Empty blocks in the table (see initial/next, minextents and freelist-related storage parameters among others) Empty space in the blocks (due to pct_free mainly) Block headers (initrans influences this size, among others) I.e. it doesn't take into account the physical storage of the data at all. dba_segments on the other hand only counts physical blocks for the segment, regardless of how much data they contain. A table with a pct_free set to 50% and never updated will have statistically half-empty blocks for example, and the calculation you did on dba_tables will reveal that when compared to dba_segments. See the Logical Storage Structures docs to learn more about how data is stored in your database (and the overhead involved).

This is a big "It Depends..." but generally speaking you don't want to have statistics updating asynchronously for most workloads because it can mean that you generate a less-than ideal execution plan based on the existing out-of-date statistics. If you have a scenario where the auto update stats execution takes an excessive amount of time and causes timeouts, you might consider going the route of letting stats update asynchronously so that it doesn't create that problem. The few seconds it typically takes to auto update statistics can easily result in minutes of difference in execution times from a better execution plan from the new statistics.

Found them. Create a table with a simple stats object. CREATE DATABASE splunge; GO USE splunge; GO CREATE TABLE dbo.foo(bar INT, munge INT); GO CREATE STATISTICS x ON dbo.foo(bar); CREATE STATISTICS y ON dbo.foo(munge); GO INSERT dbo.foo SELECT s1.[object_id], s2.[object_id] FROM sys.objects AS s1 CROSS JOIN sys.objects AS s2; GO UPDATE STATISTICS dbo.foo; GO Connect using the DAC (ADMIN:Server[\instance]). Run the following queries: DBCC SHOW_STATISTICS('dbo.foo', 'x') WITH STATS_STREAM; DBCC SHOW_STATISTICS('dbo.foo', 'y') WITH STATS_STREAM; SELECT name, imageval FROM sys.stats AS s INNER JOIN sys.sysobjvalues AS o ON s.object_id = o.objid AND s.stats_id = o.subobjid WHERE s.object_id = OBJECT_ID('dbo.foo'); You will note that imageval for each stats object isn't the same as the stats blob, but it does contain the stats blob - it's just offset. On my system it yielded this for x (I've obviously truncated a fair bit of bits): 0x0100...bunch of chars...000007000000C4E1BE00EEA0...rest the same 0x07000000C4E1BE00EEA0...rest the same And this for y: 0x0100...bunch of chars...430007000000C7E1BE00EEA0...rest the same 0x07000000C7E1BE00EEA0...rest the same The same was true for index-based statistics. You could probably do further validation of this with a series of queries using DBCC commands. First, find out the pages that are involved with the clustered index on sys.sysobjvalues (substitute your database name): DBCC IND('splunge', 'sys.sysobjvalues', 1); The result will list a bunch of pages, you're interested in the ones of PageType = 1. With a new database, you should be able to find this info on one of the pages with the highest PagePID values. E.g. on my system this was page 281, so then I looked closer at that page: DBCC TRACEON(3604); DECLARE @dbid INT = DB_ID(); DBCC PAGE(@dbid, 1, 281, 3); DBCC TRACEOFF(3604); Sure enough, I found the data in slot 17: (On larger databases, you might have to do a lot more hunting and pecking, since there's no guarantee that even a new stats object will end up on a new(er) page.) Go ahead and try this at home, but there is a reason you need to connect with the DAC for this. I'd be curious to know, of course, what you are going to do with this information that you couldn't do with DBCC SHOW_STATISTICS output. Note that this of course does not try to decode the STATS_STREAM to provide histogram or other information, and I could not find any evidence that the tabular output of DBCC SHOW_STATISTICS ... WITH HISTOGRAM is stored anywhere in table format. Joe Chang has some information about decoding if that's what you're after. I don't think it's something you'd want to do in a query - just use DBCC.

It's not recommended to use analyze to gather statistics (from 8i onward) because CBO(cost base optimizer) is build to recognize statistics that have been gathered using dbms_stats. So dbms_stats is a preferable way to gather statistics.

Try select sample_size from dba_tables where table_name='PS_JOB'; What does that return? If that number is not as expected, is it possible another DBA used something like DBMS_STATS.SET_GLOBAL_PREFS in this way for example: EXEC DBMS_STATS.set_global_prefs('estimate_percent', '20'); instead of using the default of DBMS_STATS.AUTO_SAMPLE_SIZE? (which may be more than 10 itself) That figure of 20 would change it from the traditional default of 10. However AUTO_SAMPLE_SIZE will have Oracle determine the value and that may have been set, and Oracle chose a value larger than 10. I am also in the dark about 'auto optimizer stats collection' and exactly how it has been working internally since 11g. We have some windows though (8 total) that run for longer than 3 hours of stats gathering on our production databases.

Statistic for the query planner: n_distinct You do run ANALYZE on the temporary table. You commented: The temp table is immutable: it's created once, analyzed and never changes. It exists because we had a problem with plain query being inefficient, so we extracted the rows we're interested in to a temp table to have statistics about it. Then the second query is used to find rows from dir_current which are children of those interesting ones (i.e. dir_current.parent_id=dir_process.id) dir_current is analyzed and statistics are set according to our experience (i.e. there is 10 rows on average with a given parent_id so n_distinct is set to -0.1). Given volume_id OTOH has millions of rows. But how do you do that exactly? Firstly, the rows in dir_process are the parents of the rows in dir_current (dir_current.parent_id = dir_process.id). The column relevant to your query is dir_process.id, not parent_id. Setting n_distinct for dir_process.parent_id does nothing for this query. Setting it for dir_process.id would be wrong (n_distinct is actually -1, i.e. all distinct). So that's either useless or wrong. Secondly, if done exactly like you described it (dir_current is analyzed and statistics are set) then nothing is achieved. ALTER TABLE saves the information in pg_attribute.attoptions. Only the next ANALYZE copies it to pg_statistic, which is actually consulted for query planning. Run ANALYZE after ALTER TABLE! Demo: db<>fiddle here Finally, the temp table is immutable and comparatively small. I see rows=18770 and rows=65080 for seq scan on dir_process in the plans. If that small table is reused many times it should pay to use actual counts instead of an estimate taken from the big table (ignoring that said statistic does not seem to apply to begin with). The actual data distribution in a small sample may be dramatically different. If you set n_distinct explicitly, use the actual distinct count for the current selection. Something like: DO $$ BEGIN EXECUTE format('ALTER TABLE dir_process ALTER COLUMN volume_id SET (n_distinct=%s)' , (SELECT count(DISTINCT volume_id) FROM dir_process)); END $$; But not for dir_process.id. You might set that to -1 if you don't make the column UNIQUE or PRIMARY KEY. The manual about n_distinct: If greater than zero, the estimated number of distinct values in the column. If less than zero, the negative of the number of distinct values divided by the number of rows. (The negated form is used when ANALYZE believes that the number of distinct values is likely to increase as the table grows; the positive form is used when the column seems to have a fixed number of possible values.) For example, -1 indicates a unique column in which the number of distinct values is the same as the number of rows. Related: Postgres sometimes uses inferior index for WHERE a IN (...) ORDER BY b LIMIT N Are regular VACUUM ANALYZE still recommended under 9.1? Query That aside, your query only retrieves columns from the main table. Assuming that the the temp table dir_process has no duplicates on (volume_id, id), an EXISTS semi-join is equivalent, and the predicate volume_id = ANY(volume_ids) should rather be applied to the comparatively tiny temp table (also equivalent, and statistics are up to date now!): SELECT c.id, c.volume_id, c.parent_id, c.rec_aggrs, c.blocks FROM sf.dir_current c WHERE EXISTS ( SELECT FROM dir_process p WHERE p.volume_id = ANY($volume_ids) AND p.id = c.parent_id AND p.volume_id = c.volume_id ); Index I see a UNIQUE constraint on (volume_id, id) on the main table. Since the temp table seems to be a subset, the same constraint should apply and you can replace the index on the temp table: CREATE INDEX dir_process_indx ON dir_process(volume_id, id); with a UNIQUE constraint as well: ALTER TABLE dir_process ADD CONSTRAINT dir_process_volumeid_id_unq UNIQUE (volume_id, id); (Implements a unique index on (volume_id, id) internally.) Not sure if the Postgres 9.6 query planner is helped by it, but it can only get better. More importantly, an index on (volume_id, parent_id) on the main table should be useful: CREATE INDEX dir_current_parentid_volumeid_idx ON dir_current(parent_id, volume_id); Currently there is an index on just (parent_id). But I suspect there can be many rows per parent_id, so the above should be better. (Scratch this, if parent_id is mostly unique!) The new index can mostly replace your existing index dir_current_parentid_idx on (parent_id). It's 50% bigger, that's the main downside. But you wouldn't want to maintain two indexes without good reason. See: Is a composite index also good for queries on the first field? Also, the UNIQUE constraint dir_current_volumeid_id_unq on (volume_id, id) has lost its usefulness for this query now. The "unique" aspect has always been redundant, as id is the PK and unique on its own. Unless it has other purposes, it might be removed or reduced to an appendix to the PK with the INCLUDE clause in Postgres 11 or later! Or you might make my suggested index dir_current_parentid_volumeid_idx on (volume_id, parent_id) (switched column order) instead to replace this. (And keep the index on just (parent_id), obviously.) See: Do covering indexes in PostgreSQL help JOIN columns? Does a query with a primary key and foreign keys run faster than a query with just primary keys? Should be very fast all the time now.