I have connected to my Amazon bastion and am now trying to connect to my database. However, when I try to run mysql I get an error:

PS C:\Users\antoi> mysql --u admin -p password -h127.0.0.1 -P4000
mysql : The term 'mysql' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ mysql --u admin -pfswR6c9Vbq5AvBNd4jntUCzYJaeTP_7F -h127.0.0.1 -P4306
+ ~~~~~
    + CategoryInfo          : ObjectNotFound: (mysql:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

So I have MySQL here:

PS C:\ProgramData\Microsoft\Windows\Start Menu\Programs\MySQL> dir


    Directory: C:\ProgramData\Microsoft\Windows\Start Menu\Programs\MySQL


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        27/04/2021     08:20                MySQL Installer - Community
d-----        25/06/2021     11:33                MySQL Server 5.7
-a----        14/06/2021     10:15           2238 MySQL Workbench 8.0 CE.lnk

Unfortunately there is not bin/in my directory:

I have two tables in a database, communitiesand daily_sharesI would like to make the difference of the sum of the first one with the second one:

INSERT INTO
        shares_diff (`date`, `shares_copylink`, `shares_email`, `shares_facebook`, `shares_messenger`, 
        `shares_pinterest`, `shares_twitter`, `shares_whatsapp`)
SELECT CURDATE() AS date,  
        SUM(shares_copylink) FROM communities - SUM(shares_copylink) FROM daily_shares,
        SUM(shares_email) FROM communities - SUM(shares_copylink) FROM daily_shares, 
        SUM(shares_facebook) FROM communities - SUM(shares_copylink) FROM daily_shares, 
        SUM(shares_messenger) FROM communities - SUM(shares_copylink) FROM daily_shares, 
        SUM(shares_pinterest) FROM communities - SUM(shares_copylink) FROM daily_shares, 
        SUM(shares_twitter) FROM communities - SUM(shares_copylink) FROM daily_shares, 
        SUM(shares_whatsapp) FROM communities - SUM(shares_copylink) FROM daily_shares
        WHERE EXISTS (SELECT * FROM daily_shares);

But it returns me:

You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '- SUM(shares_copylink) FROM daily_shares,
        SUM(shares_email) FROM communi' at line 5

Here is the schema .

I use MySQL 5.7

I made a table sharesin which I inserted the sum of the columns of another table communities:

CREATE TABLE communities
    (`community_id` int, `shares_copylink` int, `shares_email` int, `shares_facebook` int, `shares_messenger` int, `shares_pinterest` int, `shares_twitter` int, `shares_whatsapp` int)
;
    
INSERT INTO communities
    (`community_id`, `shares_copylink`, `shares_email`, `shares_facebook`, `shares_messenger`, `shares_pinterest`, `shares_twitter`, `shares_whatsapp`)
VALUES
    (1, 0, 0, 0, 0, 0, 0, 0),
    (2, 2, 2, 196, 0, 0, 0, 0),
    (3, 8, 20, 0, 8, 0, 0, 12),
    (6, 0, 0, 0, 0, 0, 0, 0),
    (7, 0, 0, 32, 0, 0, 0, 0),
    (8, 2, 2, 0, 1, 0, 0, 0),
    (9, 0, 0, 0, 0, 0, 0, 0),
    (12, 0, 0, 1, 1, 0, 0, 1),
    (14, 1, 1, 11, 0, 0, 0, 3),
    (16, 0, 0, 0, 0, 0, 0, 0),
    (59, 0, 0, 0, 0, 0, 0, 0),
    (60, 0, 0, 0, 0, 0, 0, 0),
    (61, 0, 0, 0, 0, 0, 0, 0),
    (62, 2, 3, 28, 0, 0, 0, 9),
    (65, 0, 0, 0, 0, 0, 0, 0),
    (66, 0, 0, 0, 0, 0, 0, 0),
    (67, 0, 0, 0, 0, 0, 0, 0),
    (68, 0, 0, 0, 0, 0, 0, 0),
    (69, 0, 0, 0, 0, 0, 0, 0),
    (70, 0, 0, 0, 0, 0, 0, 0),
    (71, 0, 0, 0, 0, 0, 0, 0),
    (72, 0, 0, 0, 0, 0, 0, 0),
    (73, 0, 0, 0, 0, 0, 0, 0),
    (74, 0, 0, 0, 0, 0, 0, 0),
    (75, 0, 0, 0, 0, 0, 0, 0),
    (76, 0, 0, 0, 0, 0, 0, 0),
    (77, 0, 0, 0, 0, 0, 0, 0),
    (78, 0, 0, 0, 0, 0, 0, 0),
    (79, 0, 0, 0, 0, 0, 0, 0),
    (80, 0, 0, 0, 0, 0, 0, 0),
    (81, 0, 0, 0, 0, 0, 0, 0),
    (82, 0, 0, 0, 0, 0, 0, 0),
    (83, 0, 0, 0, 0, 0, 0, 0),
    (84, 0, 0, 0, 0, 0, 0, 0),
    (85, 0, 0, 0, 0, 0, 0, 0),
    (86, 0, 0, 0, 0, 0, 0, 0),
    (87, 0, 0, 0, 0, 0, 0, 0),
    (88, 0, 0, 0, 0, 0, 0, 0),
    (89, 0, 0, 0, 0, 0, 0, 0),
    (90, 0, 0, 0, 0, 0, 0, 0),
    (91, 0, 0, 0, 0, 0, 0, 0),
    (93, 0, 0, 0, 0, 0, 0, 0),
    (94, 0, 0, 0, 0, 0, 0, 0),
    (95, 0, 0, 0, 0, 0, 0, 0),
    (96, 0, 0, 0, 0, 0, 0, 0),
    (97, 0, 0, 0, 0, 0, 0, 0),
    (98, 0, 0, 0, 0, 0, 0, 0),
    (99, 0, 0, 0, 0, 0, 0, 0),
    (100, 0, 0, 0, 0, 0, 0, 0),
    (101, 0, 0, 0, 0, 0, 0, 0),
    (102, 0, 0, 0, 0, 0, 0, 0),
    (103, 0, 0, 0, 0, 0, 0, 0),
    (104, 0, 0, 2, 0, 0, 0, 4),
    (105, 0, 0, 0, 0, 0, 0, 0),
    (106, 0, 0, 0, 0, 0, 0, 0),
    (107, 0, 0, 0, 0, 0, 0, 0),
    (108, 0, 0, 0, 0, 0, 0, 0),
    (109, 0, 0, 1, 0, 0, 0, 0),
    (110, 0, 0, 0, 0, 0, 0, 0),
    (111, 0, 0, 0, 0, 0, 0, 0),
    (142, 0, 0, 0, 0, 0, 0, 0),
    (143, 0, 0, 0, 0, 0, 0, 0),
    (145, 0, 1, 39, 1, 0, 0, 4),
    (146, 0, 0, 0, 0, 0, 0, 0),
    (147, 0, 0, 0, 0, 0, 0, 0),
    (148, 0, 1, 1, 1, 0, 0, 0)
;

CREATE TABLE shares (Date date, Shares int);

INSERT INTO
`shares` (`Date`, `Shares`)
SELECT
  CURDATE() as date,
  SUM(shares_copylink) + SUM(shares_email) + SUM(shares_facebook) + SUM(shares_messenger) + SUM(shares_pinterest) + SUM(shares_twitter) + SUM(shares_whatsapp) AS total
  FROM communities

On the same day I added these columns minus this result I just inserted:

SELECT CURDATE(), (SUM(shares_copylink) + SUM(shares_email) + SUM(shares_facebook) + SUM(shares_messenger) + SUM(shares_pinterest) + SUM(shares_twitter) + SUM(shares_whatsapp) - SUM(`Shares`)) AS daily_shares FROM communities, shares 

And it gives me a result other than 0 which is strange.

http://sqlfiddle.com/#!9/001e83/34/0

I use MySQL 5.7

I want to build an autocomplete form in a Django webapp. I have already been able to make the search bar in which I query my MongoDB database but how can I add an autocomplete? I have tried to adapt an official tutorial that does it with Javascript:

search_similar.html:

{% extends "todo/base.html" %}

{% block content %}

  <div class="recommendations">
    <!-- <div class="login-page"> -->
    <div class="form">
      <form class="form" action="{% url 'similar_results' %}" method="get">
        <input name="q" type="text" placeholder="Perfume name...">
        <input id="perfumename" type ="submit" value="Find Similar Perfumes"/>
      </form>
    </div>
    <script>
    $(document).ready(function() {
        $("#perfumename").autocomplete({
          source: async function(request, response){
            let data=await fetch(`http://localhost:8000/similar/similar_results?q={request.term}`)
            .then(results => results.json())
            .then(results => results.map(result => {
              return { label: result.name, value: result.name, id:result._id };
            }
            response(data);
          },
          minLength:2,
          select: function(event, ui){
            console.log(ui.item);
          }
        })
      })
    </script>
  </div>

{% endblock %}

Even though I have autocomplete="nope" the first search bar still shows chrome's default autocomplete and doesn't show the one I built in MongoDB.

I wonder if it's not a problem in the javascript, but I'm bad at javascript. Indeed the url where we arrived when we did when we press the button:

urls.py

path('similar/similar_results/', views.SearchResultsView.as_view(), name='similar_results'),

views.SearchResultsView:

class SearchResultsView(ListView):
    model = Perfume
    template_name = 'todo/search_similar_results.html'

    def get_queryset(self):  # new
        query = self.request.GET.get('q')
        print("JE SUIS PASSE PAR LA")
        # object_list = list(collection.find({"q0.Results.0.Name": {"$regex": str(query), "$options": "i"}}))
        object_list = list(collection.aggregate([
                {
                    '$search': {
                        'index': 'default',
                        'compound': {
                            'must': {
                                'text': {
                                    'query': str(query),
                                    'path': 'name',
                                    'fuzzy': {
                                        'maxEdits': 2
                                    }
                                }
                            }
                        }
                    }
                }
            ]
        ))
        print([x["name"] for x in object_list])
        return [x["name"] for x in object_list]

And it returns everything I need:

['Not a Perfume - Coffret Noël Not a Perfume', 'Festival Nite Pour Lui – Coffret Noël Eau de Toilette', ... , 'Parfum de Peau - Huile Parfumée Roll-on'

In my opinion it is possible that there is the problem in the javascript: http://localhost:8000/similar/similar_results?q={request.term}.

Update

I tried Mauricio Conteras's solution but I have problems in the backend (which I think I solve) and the frontend that I haven't solved.

back-end

I had problems with the backend. Actually I get the following error: pymongo.errors.OperationFailure: Remote error from mongot :: caused by :: autocomplete index field definition not present at path XXX. With the following pipeline:

pipeline = [
        {
            "$search": {
                "index": "default", # nombre de mi index
                "autocomplete": {
                    "path": "name", #el nombre del field sobre cual hizo el index
                    "query": str(query) # query = self.request.GET.get('q')
                }
            }
        },
        {
            "$limit": 10
        },
        {
            "$project": {
                "_id": 0,
                "title": 1
            }
        }
    ]

However, I put nombreas the path of the field definition of the index:

But with the original I return what I need:

    pipeline = [
            {
                '$search': {
                    'index': 'default',
                    'compound': {
                        'must': {
                            'text': {
                                'query': str(query),
                                'path': 'name',
                                'fuzzy': {
                                    'maxEdits': 2
                                }
                            }
                        }
                    }
                }
            },
            {
                "$limit": 10
            },
            {
                "$project": {
                    "_id": 0,
                    "name": 1
                }
            }
        ]
    result = list(collection.aggregate(pipeline))
    data = {'data': result}
    print(data)
    return JsonResponse(data)

Bring back:

{'data': [{'name': 'Not a Perfume - Coffret Noël Not a Perfume'}, {'name': 'Festival Nite Pour Lui – Coffret Noël Eau de Toilette'}, {'name': 'Festival Nite Pour Ell
e – Coffret Noël Eau de Parfum'}, {'name': "For A Kiss 'Iconic Love' - Coffret Noël Eau de Toilette"}, {'name': 'Daisy Love - Eau de Toilette'}, {'name': 'Crystal No
ir - Eau de Parfum'}, {'name': 'Rose Pompon - Eau de Toilette'}, {'name': 'Candy love - Eau de Toilette'}, {'name': 'Drakkar Noir - Eau de Toilette'}, {'name': 'Quat
re en Rose - Eau de parfum'}]}

front-end

I have tried to display the data retrieved from the backend in a simple alert:

function autocomplete(value) {
      if(!value) return;
      currentFocus = -1;
      url = `/similar/similar_results/?query=${value}`;
      fetch(url)
      .then(res => res.json())
      .then(json => {
        let data= json.data;
        alert(data);}) // <-- Aqui intento muestrar los datos que provienen del backend
      .catch(e => {
        console.error(e.message);
        console.log('Uuuuups');
      });
    }

But I never get this message.

In the console I get:

(index):99 Unexpected token < in JSON at position 2
(anonymous) @ (index):99
Promise.catch (async)
autocomplete @ (index):98
(anonymous) @ (index):140
(index):100 Uuuuups

Which seems to come from the try catch in the javascript:

Mea maxima culpa

What is returned is an html page. I think I use as endpoint what is used to return the results page when you did Enter

I have a client instance of my database and I would like to pass a database instance and the collection I want to use as a String to the client. Do you know how I can use String as attribute name?

db_url = "mongodb+srv://USER:[email protected]/a_company?retryWrites=true" \
                      "&w=majority"

Here is the function I would like to create:

def save_in_mongo(self, url, db, collection, perfume):
    client = pymongo.MongoClient(url)
    db = client.db
    fragrance = db.collection

But I don't know how to use function arguments as MongoDB client attributes .

I have a first dataframe to which I need to add the lines of a second dataframe .

It is more or less like the first one:

    QID    Questions    B   Answer1 Answer2 Answer3 F G H I J
0   3   a   4.0 a   a   a   a   e   g   i   l    
1   4   b   5.0 b   b   b   a   r   h   m   p
2   5   d   5.0 NaN e   d   b   u   e   i   z
3   6   e   5.0 d   h   r   b   c   z   i   3
...

And the second:

    QID    Questions    B   Answer1 Answer2 Answer3 F ...
0   1   a   4.0 a   a   a   a   
1   2   b   5.0 b   k   b   a   
2   2_1 z   5.0 b   k   b   a   
3   2_2 w   4.0 b   k   b   c   
4   3   d   5.0 NaN e   d   b   
5   4   e   5.0 d   h   r   b   
...

I would like to get:

    QID    Questions    B   Answer1 Answer2 Answer3 F G H I J
0   3   a   4.0 a   a   a   a   e   g   i   l    
1   4   b   5.0 b   b   b   a   r   h   m   p
2   4_1 z   5.0 b   k   b   a   r   h   m   p
3   4_2 w   4.0 b   k   b   c   r   h   m   p
4   5   d   5.0 NaN e   d   b   u   e   i   z
5   6   e   5.0 d   h   r   b   c   z   i   3
...

As you can see the dataframes share the Questions bso I have added the following lines that are included _in the new dataframe.

Literally this means that the first data box and the second data box share the same texts t1and t2in the cells of the "Answers" column. But for a given combination (t1,t2) where t1 == t2, when there are also rows below it such that QID has a _then I want to add those rows after the row they were recorded on.

Until today I tried:

rows_to_add = pd.DataFrame()
for i, row1 in df.iterrows():
  for j, row2 in df2.iterrows():
    if row1['Questions'] == row2['Questions']:
      # here I want to test if the next row has _ in his QID
      # if so I add all the lines with the same QID before _ but with row1 QID
      k = 0
      for _, next_row_df2 in df2[j+1:].iterrows():
        if "_" in str(next_row_df2['QID']):
          next_row_df2['QID'] = str(row1['QID']) + '_' + k
          rows_to_add += next_row_df2 # but I need to change the QID
        else:
          break # exit this loop and add the lines to the dataframe
        k += 1
      df = pd.concat([df.iloc[:i], rows_to_add, df.iloc[i:]]).reset_index(drop=True)
      rows_to_add = pd.DataFrame()

But it doesn't add the rows. Maybe you could do it in a more efficient way: only iterate on the df2 lines where there are _? Or with map-reduce?

Maybe you could do it in a more efficient way: only iterate on lines df2 where there is _ ? Use map reduce?

I have a docx of articles published in marketing magazines that contains scales and I would like to extract them.

For example, with the following scale I would like to obtain the title: Ten-Item and Five-Item Presonality Inventories, the questions "I see my self as:" and the answers.

{
   "title":"Ten-Item and Five-Item Presonality Inventories",
   "scales":{
      "I see myself as":{
         "answer0":"1. Extraverted, enthusiastic",
         "answer1":"2. Critical, quarrelsome",
         "answer2":"3. Dependable, self-disciplined",
         ...
      },
      "I see my self as":{
         "answer0":"Extraverted, enthusiastic (that is, sociable ... ",
         "answer1":"2. Agreeable, kind ...",
         ...
      }
   }
}

or something similar if duplicate scale names are a problem.

Until today I can extract the content separately, with the following code:

import pandas as pd

!pip install tika
from tika import parser 
!pip install python-docx 
from docx import *

# if you use colab, you need to have it in your drive in a "Books" folder
document = Document('/content/drive/My Drive/Books/handbook-of-marketing-scale-2011.docx')

dict_open = False
qas = {}
qa = {}
last_line_is_bold_or_italic = False
last_line_is_digit = False

for para in document.paragraphs:
    for run in para.runs:
        try:
            # we have a question and nothing before: begining of a q-aire 
            if run.bold or run.italic and last_line_is_digit == False: # first line must be bold # second must be numbers
                last_line_is_bold_or_italic = True
                question = run.text

            # we have an answer and last line was a question (last_line_is_bold_or_italic) or an answer (last_line_is_digit)
            # we are inserting an answer in a q-aire
            elif run.text[0].isdigit() and (last_line_is_bold_or_italic or last_line_is_digit):
                last_line_is_bold_or_italic = False
                last_line_is_digit = True
                number = run.text.split(".")[0]
                answer = run.text.split(".")[-1]
                qa[number] = answer

            # we have a question (last_line_is_bold_or_italic) and last line was an answer, we close the preceding question
            # we are inserting the q-aire in the dictionnary of all q-aires
            elif run.bold or run.italic and last_line_is_digit:
                last_line_is_digit = False
                # close dict
                qas[question] = qa
                qa = {}
                dict_open = False
              
            # we might have something bold, italics, ... but who knows what it is?
            # we reset
            else:
                last_line_is_bold_or_italic = False
                last_line_is_digit = False
                question = None
                pass

        except IndexError:
            last_line_is_bold_or_italic = False
            last_line_is_digit = False
            question = None
            if dict_open:
                   qa = {}
                   dict_open = False
            else:
                 pass


      
            

But I get:

{' ': {'11, 186–93': '',
  '2002': ' Professor Bearden served as the University SEC Faculty Athletics Representative from 2006 to 2010 and received the first Distinguished Service Award from the ',
  '23, 387–93': ''},
 ' slight agreement;': {'0, ': '0, ',
  '26 (June), 85–98': '',
  '34 (June), 456–67': ''},
 ' somewhat characteristic, ': {'1982 by the American Psychological Association': '',
  '3': '3',
  '42 (1), 116–31': ''},
 ' somewhat true, but with exceptions; ': {'4': '4'},
 ' uncertain, ': {'4': '4'},
 '.': {'05), respectively': '39 ',
  '28) was reportedfor the scale(Houseand Rizzo 1972,p': '',
  '7, 467–505': ''},
 'Academy of Marketing Science, ': {'1': ' In general, do you talk to your friends and neighbors about cable television:',
  '24 (2)': '24 (2)',
  '31, 55–64': '',
  '98\tHANDBOOK OF MARKETING SCALES': '98\tHANDBOOK OF MARKETING SCALES'},
 'Analytic/Holistic Thinking Scale: AHS': {'01)': '',
  '12, 341–52': '',
  '12, 663–82': '',
  '13, 121–37': '',
  '1995 by John Wiley & Sons, Inc': '',
  '1996 by Elsevier Science': '',
  '22, 41–53': '',
  '24–38': '',
  '288\tHANDBOOK OF MARKETING SCALES': '288\tHANDBOOK OF MARKETING SCALES',
  '51, 407–15': ''},
 'Assertiveness and Aggressiveness': {'432\tHANDBOOK OF MARKETING SCALES': '432\tHANDBOOK OF MARKETING SCALES'},
 'Attention to Social Comparison Information: ATSCI': {'128\tHANDBOOK OF MARKETING SCALES': '128\tHANDBOOK OF MARKETING SCALES',
  '224 sample': '',
  '30, 526–37': '',
  '4 ': '4 ',
  '50; and consumer behavior measures) show strong support for the validity of the ATSCI': '',
  '76, 461–71': ''},
 'Attitude Toward Private Label Products Scale': {'01) with measures of experiential shopping motives, compulsive buying, pleasure, and arousal, respec-tively': '',
  '3 (3), 239–49': '',
  '370\tHANDBOOK OF MARKETING SCALES': '370\tHANDBOOK OF MARKETING SCALES',
  '40 (August), 310–20': '',
  '5-point Likert-type scales': '',
  '7-point scales with endpoints indicated above': '',
  '79 (Summer), 77–95': '',
  '93 (college students)': ''},
 'Behavioral Identification Form: BIF': {'110 (3), 403–21': '',
  '290\tHANDBOOK OF MARKETING SCALES': '290\tHANDBOOK OF MARKETING SCALES',
  '57 (4), 660–71': ''},
 'Business Research, ': {'31': '31'},
 'Consumer Attitudes Toward Marketplace Globalization': {'27, 37–65': '',
  '29, 83–100': '',
  '392\tHANDBOOK OF MARKETING SCALES': '392\tHANDBOOK OF MARKETING SCALES',
  '6': '6'},
 'Consumer Involvement Profiles: CIP': {'18, 392–401': '',
  '1991 by University of Chicago Press': '',
  '230': '230',
  '234': '234',
  '240\tHANDBOOK OF MARKETING SCALES': '240\tHANDBOOK OF MARKETING SCALES',
  '30, 3–23': '',
  '31 (1), 3–23': '',
  '5 ': '5 ',
  '51 (July), 5–15': ''},
 'Consumer’s Need for Uniqueness: CNFU': {'44\tHANDBOOK OF MARKETING SCALES': '44\tHANDBOOK OF MARKETING SCALES',
  '58 subjects': '',
  '86 (October), 518–27': ''},
 'Customer-Based Reputation of a Service Firm: CBR Scale': {'23 (September), 227–39': '',
  '35, 127–43': '',
  '396\tHANDBOOK OF MARKETING SCALES': '396\tHANDBOOK OF MARKETING SCALES',
  '62, 924–30': ''},
 'Electronic Service Quality: E-S-QUAL': {'412\tHANDBOOK OF MARKETING SCALES': '412\tHANDBOOK OF MARKETING SCALES',
  '7 (February), 213–33': ''},
 'Emotions: Dimensions of Emotions: PAD': {'1997 by the University of Chicago': '',
  '24, 127–46': '',
  '310\tHANDBOOK OF MARKETING SCALES': '310\tHANDBOOK OF MARKETING SCALES',
  '91 (4), 780–95': ''},
 'Ethics: Improving Evaluations of Business Ethics': {'448\tHANDBOOK OF MARKETING SCALES': '448\tHANDBOOK OF MARKETING SCALES'},
 'Ethnocentrism: Consumer Ethnocentrism: CETSCALE': {'25 (1), 26–37': '',
  '28, 320–27': '',
  '92\tHANDBOOK OF MARKETING SCALES': '92\tHANDBOOK OF MARKETING SCALES'},
 'Gender Dimensions of Brand Personality': {'1997 by the American Marketing Association': '',
  '209 university students': '',
  '3': ' Overall quality of the original brand (1 ',
  '34 (August), 347–56': '',
  '34, 347–56': '',
  '342\tHANDBOOK OF MARKETING SCALES': '342\tHANDBOOK OF MARKETING SCALES',
  '346\tHANDBOOK OF MARKETING SCALES': '346\tHANDBOOK OF MARKETING SCALES',
  '4': ' Perceived difficulty in designing and making the extension (1 ',
  '46 (January), 105–19': ''},
 'General Self-Control': {'1, 2, 4, 8, and 11 compose the “hedonic” subscale': '',
  '78\tHANDBOOK OF MARKETING SCALES': '78\tHANDBOOK OF MARKETING SCALES'},
 'Horizontal and Vertical Individualism and Collectivism': {'51 (April), 407–15': '',
  '54\tHANDBOOK OF MARKETING SCALES': '54\tHANDBOOK OF MARKETING SCALES',
  '74 (1), 118–28': ''},
 'Innovativeness: Use Innovativeness': {'05) to “new product trial': '”',
  '116\tHANDBOOK OF MARKETING SCALES': '116\tHANDBOOK OF MARKETING SCALES',
  '118\tHANDBOOK OF MARKETING SCALES': '118\tHANDBOOK OF MARKETING SCALES',
  '1995 by Lawrence Erlbaum Associates, Inc': '',
  '4 (4), 329–45': ''},
 'Job Characteristic Inventory: JCI': {'20 (March), 31–44': '',
  '38 (May), 269–77': '',
  '4 ': '4 ',
  '456\tHANDBOOK OF MARKETING SCALES': '456\tHANDBOOK OF MARKETING SCALES',
  '480\tHANDBOOK OF MARKETING SCALES': '480\tHANDBOOK OF MARKETING SCALES',
  '84)': '',
  '9, 639–53': ''},
 'Journal of Consumer Research,': {'16': '16', '24)': ''},
 'Leadership: Transactional and Transformational Leadership': {'1996 by the American Marketing Association': '',
  '526\tHANDBOOK OF MARKETING SCALES': '526\tHANDBOOK OF MARKETING SCALES',
  '60, 89–105': ''},
 'Long-Term Orientation: LTO': {'01)': '',
  '10, 1–22': '',
  '16 (February), 64–73': '',
  '16 (February), 6–17': '',
  '18 (May), 133–45': '',
  '24 (4), 366–74': '',
  '26\tHANDBOOK OF MARKETING SCALES': '26\tHANDBOOK OF MARKETING SCALES',
  '28 (4), 674–89': '',
  '28 (June), 121–34': '',
  '31 (June), 209–19': '',
  '34, 100–17': '',
  '4': '4',
  '5-point scale labeled 1 ': '5-point scale labeled 1 ',
  '56 (2), 131–49': '',
  '7 (3), 309–19': '',
  '70 (1), 172–94': '',
  '78, 98–104': '',
  '88 (5), 879–903': '',
  '9 (June), 139–64': '',
  '9, 1–26': ''},
 'Meaning of Branded Products Scale': {'25, 82–93': '',
  '34, 347–56': '',
  '352\tHANDBOOK OF MARKETING SCALES': '352\tHANDBOOK OF MARKETING SCALES'},
 'Need to Evaluate: NES': {'05)': '',
  '1996 by the American Psychological Association': '',
  '38\tHANDBOOK OF MARKETING SCALES': '38\tHANDBOOK OF MARKETING SCALES',
  '5 ': '5 ',
  '70 (1), 172–94': ''},
 'Note: ': {'1 to 7': '1 to 7'},
 'Notes: ': {'1': '1',
  '15 (January), Pages 77–91': '',
  '29 (March), 551–65': ''},
 'Opinion Leadership': {'96\tHANDBOOK OF MARKETING SCALES': '96\tHANDBOOK OF MARKETING SCALES'},
 'Organizational Commitment': {'108, 17–94': '',
  '16, 321–38': '',
  '27, 333–44': '',
  '538\tHANDBOOK OF MARKETING SCALES': '538\tHANDBOOK OF MARKETING SCALES',
  '64, 295–314': ''},
 'Organizational Justice': {'540\tHANDBOOK OF MARKETING SCALES': '540\tHANDBOOK OF MARKETING SCALES'},
 'Personality and Social Psychology, ': {'01) between a measure of impulsivity and sensory innovativeness': '',
  '1990 by Elsevier Science': '',
  '20, 293–315': '',
  '42': '42'},
 'Positive and Negative Affect Scales (PANAS)': {'114 (all nonstudents)': '',
  '12, 281–300': '',
  '26 (February), 30–43': '',
  '316\tHANDBOOK OF MARKETING SCALES': '316\tHANDBOOK OF MARKETING SCALES',
  '54, 1063–70': ''},
 'Power: Dependence-Based Measure of Interfirm Power in Channels': {'01, one-tailed) indicated a reasonable degree of stability': '',
  '12, 177–87': '',
  '13': '',
  '2': '',
  '21': '',
  '34 (June), 324–40': '',
  '546\tHANDBOOK OF MARKETING SCALES': '546\tHANDBOOK OF MARKETING SCALES',
  '558\tHANDBOOK OF MARKETING SCALES': '558\tHANDBOOK OF MARKETING SCALES',
  '8': ''},
 'Pricing Tactic Persuasion Knowledge: PTPK': {'12 (December), 341–52': '',
  '382\tHANDBOOK OF MARKETING SCALES': '382\tHANDBOOK OF MARKETING SCALES'},
 'Purchasing Involvement: PI': {'1985 by the American Marketing Association': '',
  '268\tHANDBOOK OF MARKETING SCALES': '268\tHANDBOOK OF MARKETING SCALES',
  '49, 72–82': ''},
 'Reference Group Influence: Consumer Susceptibility to Reference Group Influence': {'140\tHANDBOOK OF MARKETING SCALES': '140\tHANDBOOK OF MARKETING SCALES',
  '7 (November), 1–15': ''},
 'Research, ': {'16': '', '31 (December), 551–56': '31 (December), 551–56'},
 'Response Profile: Viewer Response Profile: VRP': {'19, 37–46': '',
  '324\tHANDBOOK OF MARKETING SCALES': '324\tHANDBOOK OF MARKETING SCALES',
  '70), respectively': ''},
 'Salesperson Performance': {'1': '',
  '1993 by the American Marketing Association': '',
  '50 (1), 1–28': '',
  '512\tHANDBOOK OF MARKETING SCALES': '512\tHANDBOOK OF MARKETING SCALES',
  '520\tHANDBOOK OF MARKETING SCALES': '520\tHANDBOOK OF MARKETING SCALES',
  '57, 70–80': ''},
 'Satisfaction-Channel Satisfaction: SATIND and SATDIR': {'15), GFI ': '15), GFI ',
  '1984 by the American Marketing Association': '',
  '226–33': '',
  '32, 534–52': '',
  '35 (September), 382–97': '',
  '4 ': '4 ',
  '45 (1), 215–33': '',
  '54, 80–93': '',
  '586\tHANDBOOK OF MARKETING SCALES': '586\tHANDBOOK OF MARKETING SCALES',
  '76 (Spring), 11–32': ''},
 'Self-Concept Clarity: SCC': {'01)': '',
  '1996 by the American Psychological Association': '',
  '58\tHANDBOOK OF MARKETING SCALES': '58\tHANDBOOK OF MARKETING SCALES',
  '70 (1), 141–56': '',
  '82 Canadian students at the University of British Columbia': ''},
 'Service Convenience: SERVCON': {'35 (4), 144–56': '',
  '38 (May), 269–77': '',
  '418\tHANDBOOK OF MARKETING SCALES': '418\tHANDBOOK OF MARKETING SCALES',
  '64, 12–40': ''},
 'Service Quality of Retail Stores': {'1996 by Sage Publications': '',
  '24 (1), 3–16': '',
  '408\tHANDBOOK OF MARKETING SCALES': '408\tHANDBOOK OF MARKETING SCALES',
  '64, 12–40': ''},
 'Style of Processing Scale: SOP': {'1, 109–26': '',
  '12, 125–34': '',
  '1985 by University of Chicago Press': '',
  '296\tHANDBOOK OF MARKETING SCALES': '296\tHANDBOOK OF MARKETING SCALES',
  '36 (June), 56–72': '',
  '5 ': '5 '},
 'TV Program Connectedness Scale': {'148\tHANDBOOK OF MARKETING SCALES': '148\tHANDBOOK OF MARKETING SCALES',
  '30 (4), 526–37': ''},
 'Tension: Job-Induced Tension': {'1 ': '1 ',
  '510\tHANDBOOK OF MARKETING SCALES': '510\tHANDBOOK OF MARKETING SCALES'},
 'The Technology Readiness Index (or Techqual™)': {'122\tHANDBOOK OF MARKETING SCALES': '122\tHANDBOOK OF MARKETING SCALES',
  '2 (May),307–20': ''},
 'Value Consciousness and Coupon Proneness: VC and CP': {'30, 234–45': '',
  '386\tHANDBOOK OF MARKETING SCALES': '386\tHANDBOOK OF MARKETING SCALES'},
 'Vanity: Trait Aspects of Vanity': {'1995 by University of Chicago Press': '',
  '21, 612–26': '',
  '64\tHANDBOOK OF MARKETING SCALES': '64\tHANDBOOK OF MARKETING SCALES'},
 'Work-Family Conflict and Family-Work Conflict Scales': {'1996 by the American Psychological Association': '',
  '506\tHANDBOOK OF MARKETING SCALES': '506\tHANDBOOK OF MARKETING SCALES',
  '81 (4), 400–10': '',
  '86, CFI ': '86, CFI '},
 'exceptions; ': {'1974 by the American Psychological Association': '',
  '3': '3',
  '30 (4), 526–37': '',
  '51, 125–39': ''},
 'medium relevance': {'3, ': '3, '}}
            

I am trying to scrape questions and answers from google forms whose urls are in a csv file . Here is an excerpt:

links_y_temas.csv:

Link,Task
https://docs.google.com/forms/d/1MYSjxAMCXMXB02GPXqkewVx35_ptzv0XO7GRZQGwyGE/edit?usp=sharing,Hotel ABC
https://docs.google.com/forms/d/1VQpzX1GqsnI92J1trlP37v3GjVgNMp1h-1Rh_n8orII/edit?usp=sharing,Airline XYZ
https://docs.google.com/forms/d/1z-qyHp7O4eTI848b6L8_vh59IJ-y0bFpAo--zwPKJxY/edit?usp=sharing,Airline XYZ
https://docs.google.com/forms/d/1IoDbsif5qorINuUrF1Dl9iMtIdnAsTvVA3vMVsVHjy8/edit?usp=sharing,Airline XYZ

I'm trying to get the quiz questions and answers/urls to put into a csv file that looks like the following:

pickle_file_name,id,question,answer_1,answer_2,answer_3,answer_4,answer_5,answer_6,answer_7,answer_8,answer_9,answer_10,answer_11,answer_12,answer_13,answer_14
applicantHotel_ABC_c,1,How do you feel about your next vacation after COVID-19?,,,,,,,,,,,,,,
applicantHotel_ABC_c,2,When do you think your next vacation can start?,In next 3 months,In next 6 months,In next 1 year,Only once COVID-19 is under control,Only once COVID-19 vaccine is developed,,,,,,,,,
applicantHotel_ABC_c,3,What are your preferences regarding medical treatment policy (with additional cost)?,Doctor's availability in hotel,Ventilator availability in hotel,Tie-ups with nearby hospitals,Availability of medical rooms with primary first aid care,,,,,,,,,,
applicantHotel_ABC_c,4,What is your preferences of complementary breakfast?,Buffet breakfast with social distancing,Buffet breakfast replaced with Ala-carte with limited options,Breakfast to be delivered in room with limited options (chargeable),Packaged breakfast only,,,,,,,,,,

However, there are two types of quizzes, those that are published (the Google forms we face in real life) and those for which we have access to the backend. If I flip through both types of quizzes, I can't get the questions from the posted questions.

In effect I get the following types of exceptions:

• StaleElementReferenceException: This exception is difficult to understand. It seems to come into play when reading a posted Google Form. You can read the first question and its answers but not the rest. This can happen if a DOM operation happening on the page is temporarily causing the element to be inaccessible. To account for these cases, I'll try to access the element multiple times in a loop before finally throwing an exception.
• UnexpectedAlertPresentException: it is because it seems that there is the button to modify the location: https://docs.google.com/forms/d/1HAMUvDpYiz-SpQpKwUOxyHqn3Be7OV9vLER3K5ltrxg/edit?usp=sharing UnexpectedAlertPresentException
• content_area.get_attribute("aria-label"): I feel like I'm repeating over and over again about the same thing.

Stop talking! Here is my code that allows you to get a little over 5% of Google Forms:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
import pandas as pd
from selenium.common.exceptions import ElementNotInteractableException, NoSuchElementException
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.common import exceptions
import pickle


WDWTIME = 20
USER = '[email protected]' # ciertos google forms le requieren
PWD = "yourpassword"
def setup_chromedriver():
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    driver = webdriver.Chrome("C:\Programs\chromedriver.exe")
    """Algunos de los formularios de Google necesitan un acceso"""
    url = 'https://www.google.com/accounts/'
    driver.get(url)
    # Encontrar el login
    login_field = WebDriverWait(driver, WDWTIME).until(
        EC.presence_of_element_located((By.ID, 'identifierId')))
    login_field.send_keys(USER)
    # Haga clic en el botón siguiente
    driver.find_element_by_id('identifierNext').click()
    # Encontrar el campo de la contraseña
    time.sleep(4)
    driver.set_page_load_timeout(50)
    driver.set_script_timeout(50)
    password_field = WebDriverWait(driver, WDWTIME).until(
        EC.presence_of_element_located((By.ID, 'password')))
    password_field = password_field.find_element_by_tag_name('input')
    password_field.send_keys(PWD)
    # Haz click sobre "next" button
    driver.find_element_by_id('passwordNext').click()
    driver.set_page_load_timeout(30)
    driver.set_script_timeout(30)
    return driver


def load_data():
    df = pd.read_csv("research_assistant_intern_recruitment_an.csv")
    filter_col = ["Link"]
    return df, filter_col


def get_published_questionnaire():
    print("published questionnaire")
    questionnaire = {}
    btns = driver.find_elements_by_css_selector(".appsMaterialWizButtonEl")
    # el botón "Siguiente", *advertencia* "Solicitar acceso de edición" también está activado.
    next_btns = driver.find_elements_by_class_name("appsMaterialWizButtonPaperbuttonContent.exportButtonContent")
    if next_btns:
        next_btns[-1].click()
        next_btns = driver.find_elements_by_class_name("appsMaterialWizButtonPaperbuttonContent.exportButtonContent")
        while next_btns != []:
            containers = driver.find_elements_by_class_name(
                "freebirdFormviewerViewNumberedItemContainer"
            )
            len_containers = len(containers)
            for container in containers:
                len_containers -=1
                print("len_containers: ", len_containers)
                try:
                    question = container.find_element_by_class_name(
                        "freebirdFormviewerViewItemsItemItemTitle.exportItemTitle.freebirdCustomFont"
                    )
                except NoSuchElementException:
                    print("No question, NoSuchElementException")
                    continue
                except exceptions.StaleElementReferenceException:
                    print("No question, StaleElementReferenceException")
                    continue

                responses = container.find_elements_by_class_name(
                    "docssharedWizToggleLabeledLabelText"
                )
                extracted_text = [response.text for response in responses]
                questionnaire[question.text] = extracted_text
                content_areas = driver.find_elements_by_class_name(
                    "quantumWizTextinputSimpleinputInput.exportInput"
                )
                for content_area in content_areas:
                    skip = ["Document title", "Titre du document", "Adresse e-mail valide"]
                    if content_area.get_attribute("aria-label") in skip and not content_area.get_attribute("aria-label").isspace():
                        print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label"))
                    else:
                        print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label"))
                        content_area.send_keys("10102015")

                content_areas = driver.find_elements_by_class_name(
                    "quantumWizTextinputPaperinputInput.exportInput"
                )
                for content_area in content_areas:
                    if content_area.get_attribute("type") == "date" and not content_area.get_attribute("type").isspace():
                        condition = content_area.get_attribute("type")
                        if condition == "date":
                            content_area.send_keys("10102015")
                    elif content_area.get_attribute("max") and not content_area.get_attribute("max").isspace():
                        max = content_area.get_attribute("max")
                        content_area.send_keys(max)
                    elif content_area.get_attribute("aria-label") and not content_area.get_attribute("aria-label").isspace():
                        condition = content_area.get_attribute("aria-label")
                        print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label"))
                        if condition == "State (Two letter Abbreviation)":
                            content_area.send_keys("CA")
                        else:
                            content_area.send_keys("10102015")

                for content_area in content_areas:
                    skip = ["Document title", "Titre du document", "Adresse e-mail valide"]
                    if content_area.get_attribute("aria-label") in skip and not content_area.get_attribute("aria-label").isspace():
                        print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label"))
                    else:
                        print("content_area.get_attribute(\"aria-label\"): ", content_area.get_attribute("aria-label"))
                        content_area.send_keys("10102015")
                btns_answers = driver.find_elements_by_css_selector(".appsMaterialWizToggleRadiogroupElContainer")
                for btn_answer in btns_answers:
                    try:
                        driver.execute_script('arguments[0].scrollIntoView(true);', btn_answer)
                        btn_answer.click()
                    except ElementNotInteractableException:
                        pass
                    except exceptions.ElementClickInterceptedException:
                        continue

                # long answers
                content_areas = driver.find_elements_by_class_name(
                    "quantumWizTextinputPapertextareaInput.exportTextarea"
                )
                for content_area in content_areas:
                    content_area.send_keys("This restaurant is really good! Me and my boyfriend went there on our holiday \
                        we had dinner there at 3 of February food was 100% And the service vas 150% And i really want to thank "
                                           "\Asie for a really good service as for his coworkers. We highly recommended \
                                           this restaurant!")

                # check boxes
                btn_check_boxes = driver.find_elements_by_class_name(
                    "docssharedWizToggleLabeledContainer.freebirdFormviewerViewItemsCheckboxContainer"
                )
                for btn_check_box in btn_check_boxes:
                    btn_check_box.click()
                    break
                # btn_check_box[-1].click()

                # # other weird check boxes
                btn_check_boxes = driver.find_elements_by_class_name(
                    "docssharedWizToggleLabeledLabelText.exportLabel.freebirdFormviewerViewItemsCheckboxLabel"
                )
                for btn_check_box in btn_check_boxes:
                    btn_check_box.click()
                    break
                # btns[-1].click()
                next_btns = driver.find_elements_by_class_name(
                    "appsMaterialWizButtonPaperbuttonContent.exportButtonContent")
                if next_btns != []:
                    next_btns[-1].click()
                    next_btns = []
                else:
                    continue

    print("questionnaire: ", questionnaire)
    return questionnaire


def get_backend_questionnaire():
    print("backend questionnaire")
    # a veces empezamos con algo que parece una página publicada con un botón "siguiente"
    # if driver.find_element_by_id('identifierNext'):
    #     driver.find_element_by_id('identifierNext').click()
    questionnaire = {}
    # Obtengo todas las cartas con preguntas y respuestas dentro de ellas
    containers = driver.find_elements_by_class_name(
        "freebirdFormeditorViewItemContentWrapper"
    )
    driver.set_page_load_timeout(30)
    driver.set_script_timeout(30)
    # para cada carta
    for container in containers:
        try:
            question = container.find_element_by_css_selector(".exportTextarea[aria-label='Intitulé de la question']")
        except NoSuchElementException:
            print("NoSuchElementException in " + str(container))
            continue
        # Obtener las respuestas
        responses = container.find_elements_by_css_selector(
            ".quantumWizTextinputSimpleinputInput.exportInput"
        )
        extracted_responses = [response.get_attribute("data-initial-value") for response in responses]
        questionnaire[question.text] = extracted_responses

        driver.set_page_load_timeout(30)
        driver.set_script_timeout(30)

    print("questionnaire backend: ", questionnaire)
    return questionnaire


def extract(driver, df, survey):
    count_questionnaires = 0
    result = []
    count_not_empty = 0.0
    print("survey: ", survey)
    for location, task in zip(df.Link, df.Task):
        if task == survey:
            print("location: ", location)
            questionnaire = {}
            if "docs.google.com" in str(location):
                count_questionnaires +=1.0
                driver.get(location)
                # test if it is a published version
                try:
                    ask_access_btn = driver.find_elements_by_class_name(
                        "freebirdFormviewerViewNavigationHeaderButtonContent"
                        )
                except exceptions.UnexpectedAlertPresentException:
                    print("UnexpectedAlertPresentException")
                    get_published_questionnaire
                if ask_access_btn:
                    questionnaire = get_published_questionnaire()
                else:
                    questionnaire = get_backend_questionnaire()
            if questionnaire not in [{}, {'': ''}]:
                count_not_empty += 1.0
            print(questionnaire)
            result.append({str(count_questionnaires): questionnaire})
            count_questionnaires += 1
    print("count_questionnaires: ", count_questionnaires)
    if count_questionnaires != 0:
        print("count_not_empty/count_questionnaires: ", count_not_empty/count_questionnaires)
    return result


if __name__ == '__main__':
    """ Necesita acceder a la cuenta de Google para acceder a ciertos cuestionarios. También configurar chromedriver para que se ejecute en 
    estado sin cabeza """
    driver = setup_chromedriver()
    published_questionnaires = [] # tracking published ones
    """ Cargar CSV de los formularios de Google """
    df, columns = load_data()
    surveys = ['Hotel ABC', "Airline XYZ", "The Ministry of Tourism of France"]
    for survey in surveys:
        result = extract(driver, df, survey)
        survey = survey.replace(" ", "_")
        pickle_out = open("applicant" + survey + "_c.p", "wb")
        pickle.dump(result, pickle_out)
        pickle_out.close()
    print("published_questionnaires: ", published_questionnaires)

The output is:

C:\Users\antoi\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python3.7.exe C:/Users/antoi/Documents/Programming/Scraping/Python/new_scraper.py
survey:  Hotel ABC
location:  https://docs.google.com/forms/d/1MYSjxAMCXMXB02GPXqkewVx35_ptzv0XO7GRZQGwyGE/edit?usp=sharing
backend questionnaire
NoSuchElementException in <selenium.webdriver.remote.webelement.WebElement (session="bb78e4006674ba0bd9384f86518d8b95", element="7271d9f5-478a-4dfc-8f9e-6d9248da5bcc")>
...
NoSuchElementException in <selenium.webdriver.remote.webelement.WebElement (session="bb78e4006674ba0bd9384f86518d8b95", element="3ff21bcf-6e8d-4b88-b64d-10cf5a11b18f")>
NoSuchElementException in <selenium.webdriver.remote.webelement.WebElement (session="bb78e4006674ba0bd9384f86518d8b95", element="16e5fa0b-3502-4ab7-a7f7-a47b1b8fa388")>
NoSuchElementException in <selenium.webdriver.remote.webelement.WebElement (session="bb78e4006674ba0bd9384f86518d8b95", element="fdc1bafb-51c9-4cc7-8f43-675ea9061338")>
questionnaire backend:  {'How do you feel about your next vacation after COVID-19?': [], 'When do you think your next vacation can start?': ['In next 3 months', 'In next 6 months', 'In next 1 year', 'Only once COVID-19 is under control', 'Only once COVID-19 vaccine is developed', ''],  ... , 'Education Level': ['No higher education', 'Diploma', "Bachelor's", "Master's", 'PhD', "Other's", ''], 'Annual Income': ['< £ 30,000', '£ 30,000 to £ 50,000', '£ 50,000 to £ 80,000', '£ 80,000 to £ 120,000', '> £ 120,000', ''], 'Feedback / Comments': [''], 'Email (Optional)': ['']}
location:  https://docs.google.com/forms/d/1_iRBtfJANF5MGWqoIMQUxBdeuAa4ePMltdIsVRmdY5Y/edit?usp=sharing
published questionnaire
len_containers:  9
No question, NoSuchElementException
len_containers:  8
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
len_containers:  7
No question, StaleElementReferenceException
len_containers:  6
No question, StaleElementReferenceException
len_containers:  5
No question, StaleElementReferenceException
len_containers:  4
No question, StaleElementReferenceException
len_containers:  3
No question, StaleElementReferenceException
len_containers:  2
No question, StaleElementReferenceException
len_containers:  1
No question, StaleElementReferenceException
len_containers:  0
No question, StaleElementReferenceException
questionnaire:  {'Age': ['Under 18', '18-24', '25-34', '35-44', '45-54', 'Over 55']}
{'Age': ['Under 18', '18-24', '25-34', '35-44', '45-54', 'Over 55']}
location:  https://docs.google.com/forms/d/1j0nk_Oo-_pfJBM4UcWITDPXT97-qX5mZpb3uVyKS3CA/edit?usp=sharing
published questionnaire
len_containers:  13
No question, NoSuchElementException
len_containers:  12
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
len_containers :  ...
...
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
len_containers:  0
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
questionnaire:  {'On average, how many times per year do you travel for 2 days or more? *': ['1 to 2', '3 to 4', '5 or more'], ... , 'What are your expectations from the accommodations’ Pinterest page? (Please select all that apply) *': ['See pictures of customer service staff', 'See pictures of staff in general', 'See pictures of the destination', 'See pictures of all the types of rooms', 'See pictures of services available', 'See pictures that are not on the hotel’s website', "I don't have a Pinterest account", "I don't use Pinterest for these purposes"]}
location:  https://docs.google.com/forms/d/1kq5dhHvftF6tWmRk_7cG4H0Mkzr6xnUhpTV0j2xIYeE/edit?usp=sharing
UnexpectedAlertPresentException
published questionnaire
len_containers:  13
No question, NoSuchElementException
len_containers:  12
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
len_containers:  11
...
len_containers:  1
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
len_containers:  0
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
content_area.get_attribute("aria-label"):  Other response
questionnaire:  {'On average, how many times per year do you travel for 2 days or more? *': ['1 to 2', '3 to 4', '5 or more'],  ... 'What are your expectations from the accommodations’ Pinterest page? (Please select all that apply) *': ['See pictures of customer service staff', 'See pictures of staff in general', 'See pictures of the destination', 'See pictures of all the types of rooms', 'See pictures of services available', 'See pictures that are not on the hotel’s website', "I don't have a Pinterest account", "I don't use Pinterest for these purposes"]}
location:  https://docs.google.com/forms/d/1IFqdsm9yO8h17JsJPN4c84vpQP06PxIquWfTmRN-TVw/edit?usp=sharing

So it looks like we're lost in a loop...

I am trying to create an Efficient Frontier based simulated portfolio optimization on 50 stocks, which you can find in csv here . Now I know how to get the best portfolio but it keeps the free weights and I'm looking for an optimization that keeps this variable positive.

I get my efficient frontier by getting weights with a fixed network of volatiles σ _{p 1} ,... σ _{p n} . So for each σ _{p i} , I maximize the expected returns with the constraint that the volatility is not greater than σ _{p i} , to obtain μ _{p i} . Then σ _{p i} , μ _{p i} are n points on the efficient frontier.

In other words, we can maximize the returns r that are equal to the weights of the individual results of each action RW . This leads to the following optimization problem (I reduced it to two variables for simplicity):

Which, in matrix form, should be more or less:

Where COV is the covariance matrix between all assets.

However, I was only able to program it with no weight restrictions W .

Here, the minimization of a quadratic objective under linear constraints can be obtained by solving a linear system. Which is really easy to calculate and not that hard to understand. To get the best acuity ratio give the following:

def efficient_portfolios(returns, risk_free_rate, sigma, mu, e):
    weights_record = []
    volatilities = []
    results = np.zeros((3,len(returns)))
    i = 0
    for portfolio_return in returns:
        A = np.block([[2*sigma, mu, e], [mu.T, 0, 0], [e.T, 0, 0]])
        b = np.zeros(n+2)
        b[n] = portfolio_return
        b[n+1] = 1
        w = np.linalg.solve(A, b)[:n]
        weights_record.append(w)
        portfolio_std_dev = np.sqrt( w.T @ sigma @ w )
        volatilities.append(portfolio_std_dev)
        results[0,i] = portfolio_std_dev
        results[1,i] = portfolio_return
        results[2,i] = (portfolio_return - risk_free_rate) / portfolio_std_dev
        i+=1
    return results, weights_record, volatilities

def display_simulated_ef_with_random(mean_returns, risk_free_rate, sigma, mu, e, df):
    results, weights, volatilities = efficient_portfolios(mean_returns,risk_free_rate, sigma, mu, e)
    max_sharpe_idx = np.argmax(results[2])
    sdp, rp = results[0,max_sharpe_idx], results[1,max_sharpe_idx]
    max_sharpe_allocation = pd.DataFrame(weights[max_sharpe_idx],index=df.columns,columns=['allocation'])
    max_sharpe_allocation.allocation = [round(i*100,2)for i in max_sharpe_allocation.allocation]
    max_sharpe_allocation = max_sharpe_allocation.T

    min_vol_idx = np.argmin(results[0])
    sdp_min, rp_min = results[0,min_vol_idx], results[1,min_vol_idx]
    min_vol_allocation = pd.DataFrame(weights[min_vol_idx],index=df.columns,columns=['allocation'])
    min_vol_allocation.allocation = [round(i*100,2)for i in min_vol_allocation.allocation]
    min_vol_allocation = min_vol_allocation.T

    print("-"*80)
    print("Maximum Sharpe Ratio Portfolio Allocation\n")
    print("Annualised Return:", round(rp,2))
    print("Annualised Volatility:", round(sdp,2))
    print("\n")
    print(max_sharpe_allocation)
    print("-"*80)
    print("Minimum Volatility Portfolio Allocation\n")
    print("Annualised Return:", round(rp_min,2))
    print("Annualised Volatility:", round(sdp_min,2))
    print("\n")
    print(min_vol_allocation)

    plt.figure(figsize=(10, 7))
    plt.scatter(results[0,:],results[1,:],c=results[2,:],cmap='YlGnBu', marker='o', s=10, alpha=0.3)
    plt.colorbar()
    plt.scatter(sdp,rp,marker='*',color='r',s=500, label='Maximum Sharpe ratio')
    plt.scatter(sdp_min,rp_min,marker='*',color='g',s=500, label='Minimum volatility')
    plt.title('Simulated Portfolio Optimization based on Efficient Frontier')
    plt.xlabel('annualised volatility')
    plt.ylabel('annualised returns')
    plt.legend(labelspacing=0.8)

    return max_sharpe_allocation, min_vol_allocation

stock_prices = pd.read_csv('stocks.csv', index_col=0)
returns = stock_prices.pct_change()
mu = 252 * returns.mean().values
sigma = 252 * returns.cov().values
n = mu.shape[0]

# add risk free asset to mu/sigma
risk_free_rate = 0.0178
z = np.zeros((n,1))
#mu = np.block([mu, risk_free_rate])
#sigma = np.block([[sigma, z], [z.T, 0]])
#n = mu.shape[0]

# solve minimize w'∑w subject to μ'w = r, e'w=1 for varying r
mu = np.expand_dims(mu, axis=1)
e = np.ones((n,1))
returns = np.linspace(risk_free_rate, np.max(mu))


max_sharpe_al, min_vol_al = display_simulated_ef_with_random(returns, risk_free_rate, sigma, mu, e, stock_prices)

And gives the following plot and portfolio:

--------------------------------------------------------------------------------
Maximum Sharpe Ratio Portfolio Allocation

Annualised Return: 0.63
Annualised Volatility: 0.23


               DD  ADBE  ATVI   APD    NVS      A   ADI    AVB    AYI   AAN  \
allocation -19.33  0.03 -0.32  29.3  12.65 -14.57  2.85 -25.28 -13.17  2.77   

            ...   SWKS    NOV  KMT   MDT   RIO   PSA   STE  POWI  VALE   TX  
allocation  ... -15.61 -10.08 -7.2 -3.16  7.57 -9.39  7.93  5.13  1.07  8.4  

[1 rows x 51 columns]
--------------------------------------------------------------------------------
Minimum Volatility Portfolio Allocation

Annualised Return: 0.03
Annualised Volatility: 0.13


             DD  ADBE  ATVI   APD   NVS     A   ADI   AVB   AYI   AAN  ...  \
allocation -0.6 -7.11  5.36  3.81  22.9 -3.69  7.37 -1.27 -1.13 -0.16  ...   

            SWKS   NOV   KMT   MDT   RIO    PSA   STE  POWI  VALE    TX  
allocation  -6.4 -0.25 -9.24  6.15  4.41  19.86 -1.31 -0.23 -2.99  6.05  

But how would you do if we needed to always be positive? I know that the solution cannot be obtained simply by solving a linear system. You would need to use a quadratic optimization solver .

I tried in my code to use a primitive type as an argument to a list.

List<int> list = ... //int n’est pas un sous-type d’objet

But it seems that it is illegal et that it is necessary to use an Object

List<Integer> list = ...

Why is it not possible to use a primitive type as an argument?

I have code that is used to create 100 randomly drawn points based on a normal distribution centered at the point (1, -2) with a standard deviation of 0.3. These points are shown below.

points =  np.random.randn(100,2)*0.3 +[1,-2]
plt.scatter(points[:,0],points[:,1],c='r')
plt.show()

I want to create three groups of 200 points randomly drawn using a normal distribution of 0.05 and centered on the following centers:

• 0.8; 0.3
• 0.3 ; 0.2
• 0.5; 0.9

How to show these three groups in the same figure with three different colors?

The points generated to the previous question will be saved in a single variable x. This variable will correspond to the points of my learning set of the K-means algorithm that I want to configure.

I have an array ( list) of words that represents a tree created by SpaCy and I would like to get the first element start_VB_ROOTbut I don't get it.

>>>questionSpacy = spacy_nlp(question)
>>>treeQuestion = nltk_spacy_tree(questionSpacy)
>>>print(treeQuestion)
[Tree('start_VB_ROOT', ['When_WRB_advmod', 'did_VBD_aux', 'Beyonce_NNP_nsubj', Tree('becoming_VBG_xcomp', ['popular_JJ_acomp']), '?_._punct'])]

Indeed when it did treeQuestion[0]I get :

(start_VB_ROOT
  When_WRB_advmod
  did_VBD_aux
  Beyonce_NNP_nsubj
  (becoming_VBG_xcomp popular_JJ_acomp)

We want start_VB_ROOTbut when did treeQuestion[0][0]I getWhen_WRB_advmod

And when it did print(treeQuestion[0].label)I get:

 <bound method Tree.label of Tree('start_VB_ROOT', ['When_WRB_advmod', 'did_VBD_aux', 'Beyonce_NNP_nsubj', Tree('becoming_VBG_xcomp', ['popular_JJ_acomp']), '?_._punct'])>

So if I want to access element 1, I'm on IndexError

I have an .xlsx file and I would like to get it in the format of a MultiIndex.

I know how to get the values ​​of the first or second column.

from openpyxl import load_workbook
import pandas as pd

wb = load_workbook(filename='Trees.xlsx')

# Test if the cell values are "Division", if it is the case, store it
divisions = (cell.value for cell in wb['Industry']['A']
                 if cell.data_type == "s" and "Division" in cell.value)

# loop over the second cell, check wether there is already something in the first column
sub_divisions = (cell.value for cell in wb['Industry']['B']
                 if cell.data_type == "s")

I don't know how to fit one inside the other. I tried the following:

for cell in wb['Industry']['A']:
    if cell.data_type == "s" and "Division" in cell.value:
        divisions = divisions + cell.value
        for sub_cell in wb['Industry']['B'] # How to add the condition : from the row just below the one which had the last division
            if ((sub_cell.data_type == "s") and ("Division" not in cell['A'][cell.row+1].value)):
                sub_divisions = sub_divisions + sub_cell

It doesn't work, it's just a deletion.

I was advised by a friend to use ElastiSearch with indexes and I was also thinking of using what seems to be called SIC in the first column.

The .xlsx file is here .

I want to make a java program to replace spaces with % 20. I was thinking of making a loop and when I find a " "replacement the character with the three characters. However I have a problem with .charAt(). Indeed it seems that you can not change a character. Here is the error:

replaceSpaces.java:8: error: unexpected type
                s.charAt(i)="%20";
                        ^
  required: variable
  found:    value

Here is the entire program:

public class replaceSpaces{
    String s = "Hello Antoine";
    public static void replaceSpaces(String s){
        for(int i = 0;i<s.length();i++){
            if(s.charAt(i)==' '){
                s.charAt(i)="%20";
            }
        }
    System.out.println(s);
    }
}

I have a dictionary that has Pandas dataframes as values. I want to get a new one DataFrameby joining all the dataframes contained in the dictionary. The problem is that I need to create a new column where each row has the key value of that DataFramein the dictionary.

I think it will be better understood with an example:

We have the following dictionary, which we will call diccionario:

{0:     count  eclipse_id
    7      19       10453
    56     12       11305
    48     11       11305

 1:     count  eclipse_id
    13      9       11305
    9       5       11339}

The goal is to transform it into:

count   item_id    user_id
   19     10453          0
   12     11305          0
   11     11305          0
    9     11305          1
    5     11339          1

As can be seen, the value of each row of the column user_idcorresponds to the key that its respective DataFrame had in the original dictionary.

I have a problem in Python. I want to get the minimum index of a List:

for j in range(nbr_usines):
    L.append(Dij[j][i])
indice_min=L.index(min(L))

In the terminal it is fine with L.index(min(L)), but when I compile the program programa(Bij,Dij,Vi,Pi)it tells me:

ValueError: min() arg is an empty sequence

I want to use the cells reference (as a pointer in C++) and not their absolute values ​​in VBA in cells that vary based on one another's value.

I have done the following with Cells(x,y).Valuebut it gives me the absolute value when I want the equivalent of =$A$1$eg.

Cells(8 + i, 6 + nbrVille).Value = Worksheets("Sheet1").Cells(2, 2).Value * Worksheets("Sheet1").Cells(7, 4 + j_Distances).Value

i = 0
For i_Distancia = 1 To nbrSitio
    For j_Distancia = 1 To nbrCiudad
        Cells(8 + i, 7 + nbrCiudad).Value = ">="
        Cells(8 + i, 8 + nbrCiudad).Value = Sheets("Distancia").Cells(i_Distancia, j_Distancia ).Value - 2
        Cells(8 + i, 6 + nbrCiudad).Value = Worksheets("Sheet1").Cells(2, 2).Value * Worksheets("Sheet1").Cells(7, 4 + j_Distancia).Value
        i = i + 1

    Next j_Distancia
Next i_Distancia

That is necessary for doing dynamic constraints for a solver.

At the moment the macro gives me:

But I want :

I discovered in a C++ course for beginners that it is not necessary to use only files .cppand that we can use .hpp.

But the teacher has not explained what these files are for.

So what are headers .hppin C++ and what are they for?

A pointer gives me the address of an object, ok. Nevertheless,

• is int * x_ptr;the reverse of &?: give it a pointer and it gives me an object

• is int & x_ref = y;the reverse of *?: give it an object and give me a pointer

I have five boards

• Show( ShowID , Title, StartDate, Duration, RoomID, Singer)
• Concert( ConcertID , Date, Time, ShowID)
• Room( RoomID , Name, Address, Capacity)
• Ticket( TicketID , ConcertID, PositionNum, Category, Price)
• Sell( SaleID , SaleDate, TicketID, PaymentForm)

How to express in relational algebra people who have not performed a concert at the Liceu? (A very nice concert hall in Barcelona).

The relational algebra is a set of operations that describe step by step how to compute an answer on the relations, as they are defined in the relational model. Called procedural, unlike Relational Calculus which is declarative.

I've brought:

But I don't know if I'm expressing those who have held a concert at the Liceu to remove them from all the Cantadors.

Also, I don't know if it is necessary to write what I wrote under the ⋈, is it enough that the SalaID column of Show is also in the Rooms?