8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
Using the bq command-line tool
The bq command-line tool is a Python-based command-line tool for BigQuery. This page contains general information on using the bq command-line tool.
For a complete reference of all bq commands and ags, see bq command-line tool reference (/bigquery/docs/reference/bq-cli-reference).
Before you begin
Before you can use the bq command-line tool, you must use the Google Cloud Console to create or select a project and install the Cloud SDK.
1. Sign in (https://accounts.google.com/Login) to your Google Account.
If you don't already have one, sign up for a new account (https://accounts.google.com/SignUp).
2. In the Cloud Console, on the project selector page, select or create a Cloud project.
Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you nish these steps, you can delete the project, removing all resources associated with the project.
Go to the project selector page (https://console.cloud.google.com/projectselector2/home/dashboar
3. Install and initialize the Cloud SDK (/sdk/docs).
4. BigQuery is automatically enabled in new projects. To activate BigQuery in a preexisting project, go to Enable the BigQuery API.
Enable the API (https://console.cloud.google.com/ ows/enableapi?apiid=bigquery)
5. BigQuery provides a sandbox (/bigquery/docs/sandbox) if you do not want to provide a credit card or enable billing for your project. The steps in this topic work for a project whether or not your project has billing enabled. If you optionally want to enable billing, see Learn how to enable billing (/billing/docs/how-to/modify-project).
https://cloud.google.com/bigquery/docs/bq-command-line-tool 1/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
Instead of downloading and installing the Cloud SDK, you can use a pre-installed version of the Cloud SDK in Google Cloud Shell (/shell/docs).
General usage
Flag positioning
bq supports two kinds of ags — global ags and command ags. They should be used in the order shown here:
global_flag argument bq_command --command-specific_flag argument
Global ags (or common ags) can be used in all commands.
Command-speci c ags apply to a speci c command.
Separate multiple global or command-speci c ags using a space. For example:
bal_flag argument \ bal_flag argument \ mmand \ mand-specific_flag argument \ mand-specific_flag argument
You can specify command arguments in one of the following ways:
--flag=argument
--flag='argument'
--flag="argument"
--flag argument
--flag 'argument'
--flag "argument"
https://cloud.google.com/bigquery/docs/bq-command-line-tool 2/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
Each of these methods is used throughout the BigQuery documentation.
Some commands require the use of single or double quotes around arguments. This is often true when the argument contains spaces, commas, or other special characters. For example:
ery --nouse_legacy_sql \ CT UNT(*)
igquery-public-data`.samples.shakespeare'
Flags with boolean values can be speci ed without an argument. If you specify true or false you must specify =argument.
For example, this command speci es false for the boolean ag --use_legacy_sql by placing no at the front of the ag:
ery --nouse_legacy_sql \ CT UNT(*)
igquery-public-data`.samples.shakespeare'
Alternatively, to specify false as the ag's argument, you would enter:
ery --use_legacy_sql=false \ CT UNT(*)
igquery-public-data`.samples.shakespeare'
For a list of available global and command-speci c ags, see bq command-line tool reference (/bigquery/docs/reference/bq-cli-reference).
Ge ing help
https://cloud.google.com/bigquery/docs/bq-command-line-tool 3/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
You can enter the following commands to get help for the bq command-line tool:
For the installed version of bq, enter bq version.
For a full list of commands, enter bq help.
For a list of global ags, enter bq --help.
For help on a speci c command, enter bq help command.
For help on a speci c command plus a list of global ags, enter bq command --help.
Debugging
You can enter the following commands to debug bq:
To see requests sent and received
Add the --apilog=path_to_file ag to save a log of operations to a local le. bq works by
making standard REST-based API calls, which can be useful to see. It is also useful to attach this
log when reporting issues. Using - or stdout instead of a le path will print the log to the console. Setting --apilog to stderr outputs to the standard error le.
To help troubleshoot errors
Enter the --format=prettyjson ag when getting a job's status
(/bigquery/docs/managing-jobs#view-job) or when viewing detailed information about
resources such as tables and datasets. Using this ag outputs the response in JSON format, including the reason property. You can use the reason property to look up troubleshooting
steps (/bigquery/troubleshooting-errors).
Se ing default values for command-line ags
You can set default values for command-line ags by including them in the bq command-line tool's con guration le — .bigqueryrc. Before you con gure your default options, you must rst create a .bigqueryrc le. You can use your preferred text editor to create the le. After you create the .bigqueryrc le, you can specify the path to the le using the --bigqueryrc global ag.
https://cloud.google.com/bigquery/docs/bq-command-line-tool 4/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
If the --bigqueryrc ag is not speci ed, the BIGQUERYRC environment variable is used. If that is not speci ed, the path ~/.bigqueryrc is used. The default path is $HOME/.bigqueryrc.
Creating a .bigqueryrc le using the bq init command is not recommended.
Adding ags to .bigqueryrc
To add default values for command-line ags to .bigqueryrc:
Place global ags (/bigquery/docs/reference/bq-cli-reference#global_ ags) at the top of the le without a header
For command-speci c ags (/bigquery/docs/reference/bq-cli-reference#command-speci c_ ags) , enter the command name (in brackets) and add the command-speci c ags (one per line) below it in the following format:
nd mand-specific_flag=argument mand-specific_flag=argument
When you enter command-line ags in .bigqueryrc, you must specify the ag's argument in the following format =argument.
.bigqueryrc is read every time you run bq so changes should be updated immediately. When you run bq in interactive mode (#interactive) (bq shell), you must restart the shell for the changes to take effect.
Example
This example sets default values for the following global ags:
--apilog is set to stdout to print debugging output to the console.
--format is set to prettyjson to display command output in a human-readable JSON format.
--location is set to the US multi-region location.
https://cloud.google.com/bigquery/docs/bq-command-line-tool 5/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
This example sets default values for the following query command-speci c ags:
--use_legacy_sql is set to false to make standard SQL the default query syntax.
Note: You cannot use --nouse_legacy_sql in .bigqueryrc.
--max_rows is set to 100 to control the number of rows in the query output.
--maximum_bytes_billed is set to 10,000,000 bytes (10 MB) to fail queries that read more than 10 MB of data.
This example sets a default value for the following load command-speci c ag:
--destination_kms_key (/bigquery/docs/customer-managed-encryption) is set to projects/myproject/locations/mylocation/keyRings/myRing/cryptoKeys/myKey.
ntial_file = path_to_credential_file log=stdout mat=prettyjson ation=US
y] _legacy_sql=false _rows=100 imum_bytes_billed=10000000
] tination_kms_key=projects/myproject/locations/mylocation/keyRings/myRing/cryptoKeys/
To con rm your settings, enter the following command:
~/.bigqueryrc
Running bq in an interactive shell
You can run bq in an interactive shell where you don't need to pre x the commands with bq. To start interactive mode, enter bq shell. After launching the shell, the prompt changes to the ID of
https://cloud.google.com/bigquery/docs/bq-command-line-tool 6/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
your default project. To exit interactive mode, enter exit.
Examples
You can nd command-line examples throughout the How-to guides (/bigquery/docs/how-to) section of the BigQuery documentation. Below are links to common command-line tasks such as creating, getting, listing, deleting, and modifying BigQuery resources.
Creating resources
For information on using the bq command-line tool to create resources, see:
Creating a dataset (/bigquery/docs/datasets#bigquery-create-dataset-cli)
Creating an empty table with a schema de nition (/bigquery/docs/tables#bigquery-create-table-cli)
Creating a table from a query result (/bigquery/docs/tables#creating_a_table_from_a_query_result)
Creating an ingestion-time partitioned table (/bigquery/docs/creating-partitioned-tables#creating_an_empty_ingestion- time_partitioned_table_with_a_schema_de nition)
Creating a view (/bigquery/docs/views#creating_a_view)
For examples of creating a table using a data le, see Loading data (#loading_data).
Ge ing information about resources
For information on using the bq command-line tool to get information about resources, see:
Getting information about datasets (/bigquery/docs/dataset-metadata#getting_dataset_information)
Getting information about tables (/bigquery/docs/tables#getting_information_about_tables)
Getting information about views (/bigquery/docs/view-metadata#getting_view_information)
Listing resources
https://cloud.google.com/bigquery/docs/bq-command-line-tool 7/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
For information on using the bq command-line tool to list resources, see:
Listing datasets (/bigquery/docs/listing-datasets)
Listing tables (/bigquery/docs/tables#listing_tables_in_a_dataset)
Listing views (/bigquery/docs/listing-datasets#listing_datasets_in_a_project)
Updating resources
For information on using the bq command-line tool to update resources, see:
Updating dataset properties (/bigquery/docs/updating-datasets)
Updating table properties (/bigquery/docs/managing-tables#updating_table_properties)
Updating view properties (/bigquery/docs/updating-views)
Loading data
For information on using the bq command-line tool to load data, see:
Loading Avro data from Cloud Storage (/bigquery/docs/loading-data-cloud-storage-avro#bigquery-import-gcs- le-cli)
Loading JSON data from Cloud Storage (/bigquery/docs/loading-data-cloud-storage-json#loading_json_data_into_a_new_table)
Loading CSV data from Cloud Storage (/bigquery/docs/loading-data-cloud-storage-csv#loading_csv_data_into_a_new_table)
Loading data from a local le (/bigquery/docs/loading-data-local#loading_data_from_a_local_data_source)
Querying data
For information on using the bq command-line tool to query data, see:
Running queries (/bigquery/docs/running-queries#bigquery-query-cli)
Writing query results (/bigquery/docs/writing-results#writing_query_results)
https://cloud.google.com/bigquery/docs/bq-command-line-tool 8/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
Running parameterized queries (/bigquery/docs/parameterized-queries#running_parameterized_queries)
Using external data sources
For information on using the bq command-line tool to query data in external data sources, see:
Creating a table de nition using a JSON schema le (/bigquery/external-table-de nition#creating_a_table_de nition_using_a_json_schema_ le)
Querying Cloud Bigtable data using permanent external tables (/bigquery/external-data-bigtable#permanent-tables)
Querying Cloud Storage data using permanent external tables (/bigquery/external-data-cloud-storage#permanent-tables)
Querying Google Drive data using permanent external tables (/bigquery/external-data-drive#permanent-tables)
Expo ing data
For information on using the bq command-line tool to export data, see:
Exporting data stored in BigQuery (/bigquery/docs/exporting-data#exporting_data_stored_in_bigquery)
Using the BigQuery Data Transfer Service
For information on using the bq command-line tool with the BigQuery Data Transfer Service, see:
Setting up a Campaign Manager transfer (/bigquery-transfer/docs/doubleclick-campaign-transfer#setting_up_a_campaign_manager_transfer)
Setting up a Cloud Storage transfer (/bigquery-transfer/docs/cloud-storage-transfer#setting_up_a_cloud_storage_transfer)
Setting up a Google Ad Manager transfer (/bigquery-transfer/docs/doubleclick-publisher-transfer#setting_up_a_google_ad_manager_transfer)
https://cloud.google.com/bigquery/docs/bq-command-line-tool 9/10 8/23/2020 Using the bq command-line tool | BigQuery | Google Cloud
Setting up a Google Ads transfer (/bigquery-transfer/docs/adwords-transfer#setting_up_a_google_ads_data_transfer)
Setting up a Google Merchant Center transfer (/bigquery-transfer/docs/merchant-center-transfer#setting_up_a_google_merchant_center_transfer) (beta)
Setting up a Google Play transfer (/bigquery-transfer/docs/play-transfer#setting_up_a_google_play_transfer)
Setting up a Search Ads 360 transfer (/bigquery-transfer/docs/sa360-transfer#setting_up_a_search_ads_360_transfer) (beta)
Setting up a YouTube Channel transfer (/bigquery-transfer/docs/youtube-channel-transfer#setting_up_a_youtube_channel_transfer)
Setting up a YouTube Content Owner transfer (/bigquery-transfer/docs/youtube-content-owner- transfer#setting_up_a_youtube_content_owner_transfer)
Getting information about transfer con gurations (/bigquery/docs/working-with-transfers#getting_information_about_a_transfer_con guration)
Listing transfer con gurations (/bigquery/docs/working-with-transfers#listing_transfer_con gurations)
Viewing the run history (/bigquery/docs/working-with-transfers#viewing_the_run_history)
Viewing transfer run details and log messages (/bigquery/docs/working-with-transfers#viewing_transfer_run_details_and_log_messages)
Updating transfer con gurations (/bigquery/docs/working-with-transfers#updating_a_transfer)
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License (https://creativecommons.org/licenses/by/4.0/), and code samples are licensed under the Apache 2.0 License (https://www.apache.org/licenses/LICENSE-2.0). For details, see the Google Developers Site Policies (https://developers.google.com/site-policies). Java is a registered trademark of Oracle and/or its a liates.
Last updated 2020-07-06 UTC.
https://cloud.google.com/bigquery/docs/bq-command-line-tool 10/10