The current way to retrieve CyclObs data is to use the getData command line API. The API endpoint is https://cyclobs.ifremer.fr/app/api/getData
Notebook installation¶
conda create -n cyclobs python=3 cartopy
conda activate cyclobs
pip install xarray netcdf4 geoviews pandas rioxarray tqdm ipywidgets
wget https://gitlab.ifremer.fr/cyclobs/user_manual/-/raw/master/user_manual.ipynbTo start the notebook :
jupyter notebook
getData API usage¶
This API endpoint is used to retrieve data from our database. The endpoint is aimed to be called from command line as it returns a string in a csv-like format with comma as a separator. The getData API will not download the cyclone acquisitions for you, but it will give you files URLs where they can be downloaded and other useful information about these acquisitions (concerned cyclone, acquisition date, ...)
Various parameters (described further below) are available to modify the output format and filter the results.
Endpoint output example¶
1 - To retrieve the acquisitions for the cyclones named IDAI :
import pandas as pd
request_url = "https://cyclobs.ifremer.fr/app/api/getData?cyclone_name=IDAI"
pd.read_csv(request_url)
2 - To retrieve acquisitions from the SAR instrument for the cyclone with sid wp202019 and sh182019
The include_cols argument is used to configure the output columns
request_url = "https://cyclobs.ifremer.fr/app/api/getData?sid=wp202019,sh182019&instrument=C-Band_Synthetic_Aperture_Radar&include_cols=all"
pd.read_csv(request_url)
getData endpoint parameters¶
include_cols : comma separated list to format the csv-like output to include the given columns. Options are: cyclone_name, sid, data_url, acquisition_start_time, instrument, mission, vmax, basin. Defaults to cyclone_name,data_url
cyclone_name : commma separated list to filter wanted cyclones. Defaults to all cyclones.
sid : commma separated list to filter wanted storm id. Defaults to all storm ids.
instrument : commma separated list to filter wanted instruments. Defaults to all instruments. To see available values go to https://cyclobs.ifremer.fr/app/api/allInstruments
mission : comma separated list to filter wanted missions. Defaults to all missions. To see available values go to https://cyclobs.ifremer.fr/app/api/allMissions
basin : commma separated list to filter wanted basins. Defaults to all basins. To see available values go to https://cyclobs.ifremer.fr/app/api/allBasins
startdate : returned acquisitions returned will have acquisition start time above or equal to startdate. Format : YYYY-MM-DD. Defaults to no time limit
stopdate : returned acquisitions returned will have acquisition stop time below or equal to stopdate. Format : YYYY-MM-DD. Defaults to no time limit
cat_min : minimum category (including the catMin given limit) wanted for cyclone's acquisitions. Can be : dep, storm or cat-X with X from 1 to 5. Defaults to no category lower limit
cat_max : maximum category (excluding the catMax given limit) wanted for cyclone's acquisitions. Can be : dep, storm or cat-X with X from 1 to 5. Defaults to no category higher limit catMax must be above catMin
nopath : if set (no value needed) only the filenames will be returned in the column data_url
noheader : if set (no value needed) the csv header line will not be set in the ouput
# Retrieving the data_url for cyclone IDAI
!wget -O- "https://cyclobs.ifremer.fr/app/api/getData?sid=sh182019&include_cols=data_url&noheader"
Retrieving the data_url for cyclone IDAI and downloading the satellite acquisition files¶
wget -O- "https://cyclobs.ifremer.fr/app/api/getData?sid=sh182019&include_cols=data_url&noheader" | xargs -n1 wget -N
import geoviews as gv
import geoviews.feature as gf
gv.extension('bokeh','matplotlib')
import pandas as pd
import xarray as xr
import rasterio as rio
import rioxarray # geospatial extension for xarray
import os
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy
from tqdm.auto import tqdm
download_path="/tmp/cyclobs"
os.makedirs(download_path,exist_ok = True)
request_url="https://cyclobs.ifremer.fr/app/api/getData?sid=al122017&instrument=C-Band_Synthetic_Aperture_Radar&include_cols=all"
df_request = pd.read_csv(request_url)
# FIXME : _gd instead of _sw by default
#df_request['data_url'] = df_request['data_url'].map(lambda x : x.replace("_sw","_gd"))
# add download path
df_request['path'] = df_request['data_url'].map(lambda x : os.path.join(download_path,os.path.basename(x)))
df_request
# download 'data_url' to 'path' with wget, and read files
datasets = []
for idx,entry in tqdm(df_request.iterrows(),total=df_request.shape[0]):
ret = os.system('cd %s ; wget -N %s' % (os.path.dirname(entry['path']),entry['data_url']))
if ret == 0 :
ds = xr.open_dataset(entry['path'])
datasets.append(ds)
#datasets.append( ds.rio.reproject(projection.proj4_params))
else:
datasets.append(None) # error fetching file
df_request['dataset'] = datasets
df_request['dataset'].iloc[0]
projection=ccrs.PlateCarree()
gv_list=[gf.coastline.opts(projection=projection)]
for ds in df_request['dataset']:
gv_list.append(gv.QuadMesh(ds['wind_speed'].squeeze()[::5,::5],crs=projection).opts(cmap='jet',tools=['hover']))
gv.Overlay(gv_list).options(width=800, height=500)