I need a RegEx to leave only what is between the square brackets [ ], this is an example of the text:
HM-WP-MB-22NOV10-03[24MO+36MO+6YE+12YE RED] MPD R00
HM-WP HC-UUI MAY20 [24M+36M+6Y+12U]
You must provide the following for this example:
[24MO+36MO+6YE+12YE RED]
[24M+36M+6Y+12U]
It only works what is inside the [ ]
Dear, I am developing some queries and I want to have a group which between dates for each 'pfr_Fault_Code'there are no more than 30 days.
If they run the following code, they can see a grouping 'pfr_Fault_Code'by date and a count of events:
from collections import defaultdict
import pandas as pd
from datetime import datetime, timedelta
from time import time
df = pd.read_csv('FDE_before_delay_aviso_previo.csv', header=0, sep=',', usecols=[1,16,20,27,28],parse_dates = ['fault_date'])
df2=df.sort(columns=['CASS_ID','pfrs_pfr_date','fault_date'])
df2 = df2.reset_index(drop=True)
df2=df2.groupby(['pfr_Fault_Code','fault_date']).count()
print(df2)
I want to count the same, but as long as there are no more than 30 days in each date for each 'pfr_Fault_Code'from the most current date backwards . On the other hand, the dates are ordered. In pandas will it be possible to do this directly?
Example:
For one 'pfr_Fault_Code'=XXXXXXI have the following'fault_date'=[2017/12/01],[2017/11/29],[2017/11/10],[2017/09/30],[2017/09/15]
From the most current date back I see if there is a cut between dates greater than 30 days, which is generated between '[2017/11/10]and [2017/09/30]therefore it works for me 'fault_date'=[2017/12/01],[2017/11/29],[2017/11/10].
I was thinking of adding something like this, a new column with the difference in days, but it doesn't work for me:
df2['diferencia'] =df2.groupby('pfr_Fault_Code')['fault_date'].transform(pd.Series.diff).fillna(df2['fault_date'])
TypeError: unsupported operand type(s) for -: 'str' and 'str'
The CSV file to be able to execute the code is the following:
https://drive.google.com/file/d/1f8IFNvMA0Zbm_t0whweZjt4QSHR-ahVy/view?usp=sharing
I am performing a data transformation and I need to run a whilebetween date periods, but I have the error:
"TypeError: Cannot compare type 'Timestamp' with type 'str'"
I have seen type()what type of data they are and both are class 'pandas.tslib.Timestamp'therefore I do not understand what happens.
I am attaching my full code for you to see and a CSV file for you to run and test:
from collections import defaultdict
import pandas as pd
from datetime import datetime, timedelta
from time import time
fecha_i=[]
fecha_f=[]
contador=[]
id=[]
contador={}
contador = defaultdict(list)
df = pd.read_csv('contour-export-2017-12-14.csv', header=0, sep=',',parse_dates = ['FCH_HORA_INICIO'],dayfirst = True, usecols=[0,3,6,7])
fecha_i=df['FCH_HORA_INICIO']
fecha_f=df['FCH_HORA_TERMINO']
id=df['ID']
acumulado=fecha_i[0]
i=1
k=0
print(type(acumulado))
print(type(fecha_i[0]))
while(acumulado<datetime.now()):
acumulado=fecha_i[0]+timedelta(days=i)
k=0
while k<=len(df)-1:
if acumulado>=fecha_i[k] and acumulado<=fecha_f[k]:
contador[acumulado].append(str(ID[k]))
k=k+1
else:
k=k+1
i=i+1
File: https://drive.google.com/file/d/18sKmsg9MSs_t1JWKRYArxsVP4OPud4jL/view?usp=sharing
I am uploading very large CSV files where there is information that I do not use. What I currently do is pass to lists and then filter with conditions.
For example, in one column I have "A" and "B" but I only want those rows that have "A", so I filter out the "B's" and do it for all other lists that I load from Pandas. I think it's very inefficient what I'm doing, of loading with Pandas and then converting the columns to lists to filter them.
Is there a more efficient way to filter the data. I load the data like this in Pandas:
import pandas as pd
df = pd.read_csv('C:\\Users\\4209456\\Downloads\\TECNICO\\BBDD FD.csv', header=0, sep=';',parse_dates = ['Date'],dayfirst = True)
and then I create lists of some columns:
FD=df["FD Text"]
Fecha=pd.to_datetime(df["Date"],dayfirst = True)
Hence for both filter lists according to a condition of the list "FD", creating another list "FD2", within the condition I have the list "Date" creating a corrected list "Date2". This way I have the lists I need to start my code, which are "FD2" and "Date2", keeping the original positions of the data left over from the filter.
It's not a code question, basically I haven't been able to find anything similar to this.
I did a development in Python that I have to run on a desktop computer, but the problem is that this computer does not have an internet connection and I need to have the Pandas package installed. Is it possible to install Pandas without having internet, carrying it downloaded on a flash drive for example?
I'm programming a little development, but I have problems writing to a CSV I don't want panda to write the name of the fields, but I couldn't remove this option, I read the documentation and added the argument "index_label=False" and "index=False " when exporting the file as follows:
import pandas as pd
a=[1,2,3,4,5]
b=[6,7,8,9,10]
expHISTORIA=pd.DataFrame([a,b]).transpose() #crear DataFrame
expHISTORIA.to_csv('Historias.csv',index=False,index_label=False, mode='a',sep=';') #crear el CSV
The problem is that the generated file contains fields, I want it not to print them, it only generates the file with the data.
What it prints is the following:
0 1
1 6
2 7
3 8
4 9
5 10
But the first two numbers are (0, 1) are fields that are created automatically and I don't want them to be there. Ideally it would print this:
1 6
2 7
3 8
4 9
5 10
I have several lists and I want to create only one Dataframe to later export.
The lists are of the type:
a=[1, 2, 3, 4, 5, 6, 7, 8]
b=[9, 10, 11, 12, 13, 14, 15, 16]
For this I am creating 2 dataframes as follows:
export=pd.DataFrame(a, columns=['A'])
export2=pd.DataFrame(b, columns=['B'])
My problem is that I have many lists and I would like to make only one Dataframe like this:
export=pd.DataFrame(a, b, columns=['A', 'B'])
But it throws me an error, of dimension:
ValueError: Shape of passed values is (1, 600), indices imply (2, 576)
I'm importing data from a Google Sheets spreadsheet, but I need to filter some rows and not load all of them. Does anyone run this library, and know if there is an argument to filter rows.
I am doing it as follows so far:
import gspread
from oauth2client.service_account import ServiceAccountCredentials
scope=['https://spreadsheets.google.com/feeds']
creds=ServiceAccountCredentials.from_json_keyfile_name('Python- cbe875b7fd9f.json', scope)
client=gspread.authorize(creds)
sheet=client.open('Prueba Python').sheet1
preventivos=sheet.get_all_records()
print(preventivos)
I do this because there is a column that has several options for example "NOGO", "MOC" and "AHM" option. I want to import only those rows that have the "AHM" option. In this way the load will be faster since now it takes several seconds and there is a lot of data that I am not interested in importing.
Example file:
https://docs.google.com/spreadsheets/d/1PWKujfqoMpdecMWNxGRxPQoA4RtmZeMVBemH6rXfVF8/edit#gid=0
In a part of my code I ask a question on the screen. The question as an answer requires a "yes" or "no", but when they misspell or something else I would like the code to return and ask again until the user correctly enters the option.
What I am doing is the following:
print("RESPONDA con si o no")
clave=input()
if clave=="N" or clave=="NO" or clave=="no" or clave=="No" or clave=="n" or clave=="nO":
print("ok")
Does anyone know how to fix this? I can come up with a whilethat is in the loop as long as it responds fine, ending this with a break.
Dear, does anyone know a method to find non-exact matches between text strings?
For example:
I have the following text "STATUS MSG PACK ACM L" (column 1) and it should return "PACK L" (column 2).
I have 2 lists, one written by a person that are longer texts and another that corresponds to the message to search for, which is the correct one.
I enclose an example of the two lists: column 1 should be searched for in column 2, and return the most associated element of column 2:
https://drive.google.com/file/d/0B11sJdX_AaJBd2lvWGszaFpXM2c/view?usp=sharing
Dear, I have several lists that depend on each other, for example in one list I have a code and in another I have the date of that code. My problem is that I had to add a new list of codes that just as has your list of dates. My intention is to create a single list of dates ordered chronologically but keeping the position relative to its list of codes.
What I was doing was pasting the new list of dates and the new list of codes at the end, leaving a single large list of codes and another large list of dates, like this:
for i in range(len(codigo2)): #Union lista principal FDE, Fecha y Status
CODIGOS.append(codigo2[i])
FECHAS.append(fechas2[i])
But by doing this the chronological order is lost since I am only adding the new codes and dates to the end. They know a way to sort a list and keep those new positions for another list.
Hello, I have a list of decimals that I converted to String, to be able to change the points for commas. But with "replace" it doesn't allow me to make the change, I'm doing it this way:
for i in range(len(UCL)): #UCL de corte
UCL[i].replace(".",",")
print(UCL[i])
The original UCL list is of the form:
['2.363636368', '2.17391304', '3.875', '2.282828286', '2.383838386', '2.616161618', '2.363636368', '4.85', '4.75', '2.282828289', '2.616161618', '0.666666666', '3.0', '3.272727272', '2.434343439', '0.666666666', '0.666666666', '2.538461538', '0.666666666', '3.875', '2.616161618', '2.393939396', '0.666666666', '0.666666666', '0.666666666', '0.666666666', '2.383838384']
My idea is to print the same thing but change the dots to commas inside each string of the UCL list.
I am programming a task in Python but when loading the dates from a CSV using Pandas it loads them in an erroneous way, sometimes it changes the day for the month. For example, for these cases they should all be from the month 06, but sometimes it is 06 and other times it exchanges 06 for the day, changing the month, for example row 19 to 23, 27 and 28:
......
19 2017-03-06 01:10:00
20 2017-03-06 01:10:00
21 2017-03-06 17:44:00
22 2017-03-06 17:44:00
23 2017-04-06 04:12:00
24 2017-06-06 04:21:00
25 2017-06-06 04:21:00
26 2017-06-06 15:37:00
27 2017-09-06 18:43:00
28 2017-09-06 18:43:00
29 2017-09-06 21:59:00
....
I load the file like this and create a list of dates:
df = pd.read_csv('PORFA.csv', header=0, sep=';')
Date2=pd.to_datetime(df["FDE Date"]) #Crear lista de Fechas
But when printing "Date2" it throws it as above.
I attach the BBDD that I am reading:
https://drive.google.com/file/d/0B11sJdX_AaJBRllVVjdkenE3RzA/view?usp=sharing
I am programming a development and I want to compare 2 dictionaries, which in common have keys of the same name.
For example dictionary 1 is of the form:
dic1 = {'0': [2.4636363650000002], '6': [4.1666666650000002], '9': [4.8333333349999998], '11': [3.5000000090000012], '14': [6.6181818249999989]}
Dictionary 2 is of the form:
dic2 = {'0': [2, 3, 4], '6': [5,6,7], '19': [4.8333333349999998], '10': [4.8333333349999998], '12': [3.5012]}
Seeing, dictionary 2 matches the 1 keys '0' and '6'.
My idea is to create a dictionary 3 containing only this:
dic3 = {'0': [2, 3, 4], '6': [5,6,7]}
What I am doing is creating lists with the keys of each dictionary, then I compare these lists and in the new dictionary I add those elements of the keys that match. I want to know if you know of a faster and more efficient way than looping through and creating lists, since there are so many keys I have.
I need to write to the same CSV every day, but every time I compile it rewrites the file from scratch. My idea would be to read the number of lines that the file has written and write the new information in that row.
I am currently doing it this way:
creacion = {0: [0.081818182000000003], 2: [0.30769230800000003], 8: [0.0]})
my_df = pd.DataFrame(creacion)
my_df.to_csv('out.csv', index=False, header=False,sep=';',decimal=',')
The output I get from the CSV is:
0,081818182;0,307692308;0
But if I compile again other information rewrites it from scratch:
creacion = {3: [0.0], 4: [0.22222], 5: [0.1]})
my_df = pd.DataFrame(creacion)
my_df.to_csv('out.csv', index=False, header=False,sep=';',decimal=',')
The output I get from the CSV is:
0;0,22222;0,1
The acceptable output would be:
0,081818182;0,307692308;0
0;0,22222;0,1
The other thing, could you also add the dictionary key directly?
What I want to do ideally is this:
0;0,081818182
2;0,307692308
8;0
3;0
4;0,22222;0,1
5;0,1
I need to find the most efficient way to remove duplicates from a list in Python.
I am doing it this way:
for i in mj:
if i not in mj2:
mj2.append(i)
where kjis a list like [2, 4, 4, 4, 4, 4, 9, 9]and the output mj2is of the form:
[2, 4, 9]
There is a more efficient way that doesn't include loops, since I have to parse large lists.
I need to create a dictionary that has the sum of elements of another dictionary, in the most computationally efficient way.
The main dictionary is of the form:
{'11': [0.76700000000000002, 3455.0, 0.76700000000000002, 0.76700000000000002, 34.0, 0.76700000000000002], '22': [34.0, 0.76700000000000002]}
I need to create a new dictionary containing the same keys but with the sum of the integers from the first dictionary. The output would then be:
{'11': [3458], '22': [34.767]}
I want to add new items to a dictionary in a for loop, but it gets replaced. I'm adding it like this:
for j in range(len(Repetidos)):
historias[m[j]]=Repetidos[j]
Is it okay how I do it? since in the end it throws me only one element for each dictionary key, the idea is that each key contains several elements. Or should it be done with "append"
Cheers!
I'm programming in Python and reading from a CSV template with Numpy like this:
csv = np.genfromtxt('MMRExport.csv',delimiter=",", dtype=str)
But inside the CSV I have rows of empty data and it doesn't take them into account, for example the first row has:
"A","B","C","","D","E",""
Then the columns that have empty fields are not considered or it throws me an error in the cases that following rows have data in that location. For example, if row 2 is the same shape as row 1 there is no problem, but if another row was of the form:
"1","","2","3","4","5","6"
It throws me an error since the number of columns are different. What I want is to read all the columns even the empty ones. How could I do that?
The error it throws me is this:
Traceback (most recent call last):
File "/Users/JPontigo/Downloads/Tarea.py", line 17, in <module>
csv = np.genfromtxt('MMRExport.csv',delimiter=",", dtype=str)
File "/Users/JPontigo/anaconda/lib/python3.5/site- packages/numpy/lib/npyio.py", line 1769, in genfromtxt
raise ValueError(errmsg)
ValueError: Some errors were detected !
Line #2 (got 27 columns instead of 5)
Line #3 (got 27 columns instead of 5)
Line #4 (got 27 columns instead of 5)
Line #5 (got 27 columns instead of 5)
Line #6 (got 2 columns instead of 5)
Line #7 (got 27 columns instead of 5)
Line #8 (got 27 columns instead of 5)
Line #9 (got 27 columns instead of 5)
Line #10 (got 27 columns instead of 5)
The file is this:
https://drive.google.com/file/d/0B11sJdX_AaJBWGNabERCMzhiYVk/view?usp=sharing
I am doing a little development, and I need to have the position of the repeated elements of a list.
What I have done for this is the following:
FDE = [1,2,3,4,4,3]
for k in range(6):
m=[i for i,x in enumerate(FDE) if x==FDE[k]]
print(m)
The result that the screen shows me is:
m=[2.5]
In short, it only looks for the repeated one '3'but not the '4'one that is also repeated , ideally it would print :
m=[2,3,4,5]