Using Python 3.6.9, I'm making requests with Requests by calling URLs from a text file. I put five of them here, for example. The URLs are all valid. https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML= 2001 Gardner Cir W Aurora IL https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML= 2427 Oakfield Dr Aurora IL https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML= 2451 Avalon Ct Aurora IL https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML= 2516 Hillsboro Blvd Aurora IL https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML= 2623 Shenandoah Ct Aurora IL What I intend it to do is iterate the list of URLs and request each one in the for loop. Then it prints out the BeautifulSoup object. It looks like an XML when it's printed which is what I want. While debugging, I see it loops once and prints the first URL as desired, but then throws an error at the top of the for loop. I don't see what's wrong with the URL? ''' (1) putting the URLs one by one in the browser, (2) get the resulting XMLs, and (3) listing these XMLs as text in a .txt file ''' import requests from bs4 import BeautifulSoup txtfile = r 'C:\Users\jpilbeam\USPSAPIWCHDUpdateAll.txt' #convert text file into a list with open ( txtfile ) as f : x = ( list ( map ( str . strip , f . readlines ( ) ) ) ) for i in x : #Request the URL response = requests . get ( i ) #see if the URL has been correctly encoded print(r.url) r_url = response . text #parse the downloaded homepage to get a beautifulsoup object new_xml = BeautifulSoup ( r_url , features = "xml" ) . prettify ( ) print ( new_xml ) Error: >>> [Dbg]>>> 2001 GARDNER CIR W AURORA IL 60503 6213 Traceback (most recent call last): File "\\gisfile\GISstaff\Jared\Python Scripts\ArcGISPro\CallToFromUSPS_II.py", line 27, in response = requests.get(i) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py", line 75, in get return request('get', url, params=params, **kwargs) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py", line 519, in request prep = self.prepare_request(req) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py", line 462, in prepare_request hooks=merge_hooks(request.hooks, self.hooks), File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py", line 313, in prepare self.prepare_url(url, params) File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py", line 387, in prepare_url raise MissingSchema(error) requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://? >>>

nav[aria-label="Primary Navigation"] { padding: 0; & ul { list-style: none; width: 100%; display: flex; flex-direction: row; justify-content: start; align-items: start; gap: 30px; padding: 0; & li { margin: 0; } & ul li { list-style: none; } } }

MissingSchema: Invalid URL ' ': No schema supplied

jpilbeam

Using Python 3.6.9, I'm making requests with Requests by calling URLs from a text file. I put five of them here, for example. The URLs are all valid.

https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID="564WILLC0589"><Address><Address1>2001 Gardner Cir W</Address1><Address2></Address2><City>Aurora</City><State>IL</State><Zip5></Zip5><Zip4></Zip4></Address></AddressValidateRequest>

https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID="564WILLC0589"><Address><Address1>2427 Oakfield Dr</Address1><Address2></Address2><City>Aurora</City><State>IL</State><Zip5></Zip5><Zip4></Zip4></Address></AddressValidateRequest>

https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID="564WILLC0589"><Address><Address1>2451 Avalon Ct</Address1><Address2></Address2><City>Aurora</City><State>IL</State><Zip5></Zip5><Zip4></Zip4></Address></AddressValidateRequest>

https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID="564WILLC0589"><Address><Address1>2516 Hillsboro Blvd</Address1><Address2></Address2><City>Aurora</City><State>IL</State><Zip5></Zip5><Zip4></Zip4></Address></AddressValidateRequest>

https://secure.shippingapis.com/ShippingAPI.dll?API=Verify&XML=<AddressValidateRequest USERID="564WILLC0589"><Address><Address1>2623 Shenandoah Ct</Address1><Address2></Address2><City>Aurora</City><State>IL</State><Zip5></Zip5><Zip4></Zip4></Address></AddressValidateRequest><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

What I intend it to do is iterate the list of URLs and request each one in the for loop. Then it prints out the BeautifulSoup object. It looks like an XML when it's printed which is what I want. While debugging, I see it loops once and prints the first URL as desired, but then throws an error at the top of the for loop. I don't see what's wrong with the URL?

<SPAN class="string token">''' (1) putting the URLs one by one in the browser, (2) get the resulting XMLs,
 and (3) listing these XMLs as text in a .txt file '''</SPAN>


<SPAN class="keyword token">import</SPAN> requests
<SPAN class="keyword token">from</SPAN> bs4 <SPAN class="keyword token">import</SPAN> BeautifulSoup


txtfile <SPAN class="operator token">=</SPAN> r<SPAN class="string token">'C:\Users\jpilbeam\USPSAPIWCHDUpdateAll.txt'</SPAN>

<SPAN class="comment token">#convert text file into a list</SPAN>
<SPAN class="keyword token">with</SPAN> open <SPAN class="punctuation token">(</SPAN>txtfile<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> f<SPAN class="punctuation token">:</SPAN>
    x <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">(</SPAN>list<SPAN class="punctuation token">(</SPAN>map<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">.</SPAN>strip <SPAN class="punctuation token">,</SPAN>f<SPAN class="punctuation token">.</SPAN>readlines<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>

    <SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> x<SPAN class="punctuation token">:</SPAN>
        <SPAN class="comment token">#Request the URL</SPAN>
        response <SPAN class="operator token">=</SPAN> requests<SPAN class="punctuation token">.</SPAN>get<SPAN class="punctuation token">(</SPAN>i<SPAN class="punctuation token">)</SPAN>
        <SPAN class="comment token">#see if the URL has been correctly encoded print(r.url)</SPAN>
        r_url <SPAN class="operator token">=</SPAN> response<SPAN class="punctuation token">.</SPAN>text

        <SPAN class="comment token">#parse the downloaded homepage to get a beautifulsoup object</SPAN>
        new_xml <SPAN class="operator token">=</SPAN> BeautifulSoup<SPAN class="punctuation token">(</SPAN>r_url<SPAN class="punctuation token">,</SPAN> features <SPAN class="operator token">=</SPAN> <SPAN class="string token">"xml"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>prettify<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
        <SPAN class="keyword token">print</SPAN><SPAN class="punctuation token">(</SPAN>new_xml<SPAN class="punctuation token">)</SPAN><SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Error:

>>> 
[Dbg]>>> 
<?xml version="1.0" encoding="utf-8"?>
<AddressValidateResponse>
 <Address>
  <Address2>
   2001 GARDNER CIR W
  </Address2>
  <City>
   AURORA
  </City>
  <State>
   IL
  </State>
  <Zip5>
   60503
  </Zip5>
  <Zip4>
   6213
  </Zip4>
 </Address>
</AddressValidateResponse>
Traceback (most recent call last):
  File "\\gisfile\GISstaff\Jared\Python Scripts\ArcGISPro\CallToFromUSPS_II.py", line 27, in <module>
    response = requests.get(i)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py", line 519, in request
    prep = self.prepare_request(req)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\sessions.py", line 462, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py", line 313, in prepare
    self.prepare_url(url, params)
  File "C:\Program Files\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\requests\models.py", line 387, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL '': No schema supplied. Perhaps you meant http://?
>>> <SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Find more posts tagged with

Accepted answers

JoshuaBixby

It is saying you have an invalid URL, and empty URL in this case. If the URLs you posted here came from the text file, then you appear to have an extra LF or CR creating a space between them, which would be read as an empty line. Trying just printing the URLs in the loop to make sure they are getting read correctly before passing them to requests.

All comments

JoshuaBixby

jpilbeam

Yes, that's exactly how they look in the text file. I just looped through the text file without passing to requests and they were read correctly. They look fine. But, good call on the spaces between the URLs. I'll take the spaces out and try again.

jpilbeam

That was it! I took out the white spaces in the text file and then ran it again. It now prints all the URLs.

For some reason, though, it's only writing the very last URL to file? The write function is not in the for loop, so why would it not be writing all the URLs?

<SPAN class="comment token">#convert text file into a list</SPAN>
<SPAN class="keyword token">with</SPAN> open<SPAN class="punctuation token">(</SPAN>txtfile<SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> f<SPAN class="punctuation token">:</SPAN>
    x <SPAN class="operator token">=</SPAN> <SPAN class="punctuation token">(</SPAN>list<SPAN class="punctuation token">(</SPAN>map<SPAN class="punctuation token">(</SPAN>str<SPAN class="punctuation token">.</SPAN>strip <SPAN class="punctuation token">,</SPAN>f<SPAN class="punctuation token">.</SPAN>readlines<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">)</SPAN>
    <SPAN class="keyword token">for</SPAN> i <SPAN class="keyword token">in</SPAN> x<SPAN class="punctuation token">:</SPAN>
        <SPAN class="comment token">#Request the URL</SPAN>
        response <SPAN class="operator token">=</SPAN> requests<SPAN class="punctuation token">.</SPAN>get<SPAN class="punctuation token">(</SPAN>i<SPAN class="punctuation token">)</SPAN>
        <SPAN class="comment token">#see if the URL has been correctly encoded print(r.url)</SPAN>
        r_url <SPAN class="operator token">=</SPAN> response<SPAN class="punctuation token">.</SPAN>text

        <SPAN class="comment token">#parse the downloaded homepage to get a beautifulsoup object</SPAN>
        new_xml <SPAN class="operator token">=</SPAN> BeautifulSoup<SPAN class="punctuation token">(</SPAN>r_url<SPAN class="punctuation token">,</SPAN> features <SPAN class="operator token">=</SPAN> <SPAN class="string token">"xml"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>prettify<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>


<SPAN class="comment token">#write new list to file in your directory</SPAN>
<SPAN class="keyword token">with</SPAN> open<SPAN class="punctuation token">(</SPAN>r<SPAN class="string token">'C:\Users\jpilbeam\USPSAPIWCHDUpdateAll_II.txt'</SPAN><SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"w"</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> api_list<SPAN class="punctuation token">:</SPAN>
    api_list<SPAN class="punctuation token">.</SPAN>write<SPAN class="punctuation token">(</SPAN>new_xml<SPAN class="punctuation token">)</SPAN>
    api_list<SPAN class="punctuation token">.</SPAN>close‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

JoshuaBixby

You need your write statement within the for loop. the way the code is currently written, by the time the write method called, new_xml only contains the last URL information.

Maybe something like:

f_in <SPAN class="operator token">=</SPAN> <SPAN class="comment token"># path to input file</SPAN>
f_out <SPAN class="operator token">=</SPAN> <SPAN class="comment token"># path to output file</SPAN>

<SPAN class="keyword token">with</SPAN> open<SPAN class="punctuation token">(</SPAN>file_in<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"r"</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> f_in<SPAN class="punctuation token">,</SPAN> open<SPAN class="punctuation token">(</SPAN>file_out<SPAN class="punctuation token">,</SPAN> <SPAN class="string token">"w"</SPAN><SPAN class="punctuation token">)</SPAN> <SPAN class="keyword token">as</SPAN> f_out<SPAN class="punctuation token">:</SPAN>
    <SPAN class="keyword token">for</SPAN> url <SPAN class="keyword token">in</SPAN> f_in<SPAN class="punctuation token">:</SPAN>
        <SPAN class="comment token">#Request the URL</SPAN>
        response <SPAN class="operator token">=</SPAN> requests<SPAN class="punctuation token">.</SPAN>get<SPAN class="punctuation token">(</SPAN>url<SPAN class="punctuation token">)</SPAN>
        <SPAN class="comment token">#see if the URL has been correctly encoded print(r.url)</SPAN>
        r_url <SPAN class="operator token">=</SPAN> response<SPAN class="punctuation token">.</SPAN>text

        <SPAN class="comment token">#parse the downloaded homepage to get a beautifulsoup object</SPAN>
        new_xml <SPAN class="operator token">=</SPAN> BeautifulSoup<SPAN class="punctuation token">(</SPAN>r_url<SPAN class="punctuation token">,</SPAN> features <SPAN class="operator token">=</SPAN> <SPAN class="string token">"xml"</SPAN><SPAN class="punctuation token">)</SPAN><SPAN class="punctuation token">.</SPAN>prettify<SPAN class="punctuation token">(</SPAN><SPAN class="punctuation token">)</SPAN>
        
        <SPAN class="comment token">#write new list to file in your directory</SPAN>
        f_out<SPAN class="punctuation token">.</SPAN>write<SPAN class="punctuation token">(</SPAN>new_xml<SPAN class="punctuation token">)</SPAN>
<SPAN class="line-numbers-rows"><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN><SPAN>‍</SPAN></SPAN>

Join the Conversation

Signed in members can post, follow updates, and more. New here? Register a free account.

Quick Links

Community Help
Find useful guides, FAQs, and documents to help you navigate and make the most of Esri Community.