Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storage emulator returns 403 Forbidden when trying to access files uploaded with firebaseStorageDownloadTokens set in metadata #3396

Closed
miikebar opened this issue May 22, 2021 · 1 comment · Fixed by #3403
Closed

Storage emulator returns 403 Forbidden when trying to access files uploaded with firebaseStorageDownloadTokens set in metadata #3396

miikebar opened this issue May 22, 2021 · 1 comment · Fixed by #3403
Assignees
@abeisgoat
Labels
emulator-suite emulators: storage type: bug

Comments

@miikebar
Copy link

miikebar commented May 22, 2021 •
edited
Loading

[REQUIRED] Environment info

firebase-tools: 9.11.0

Platform: Windows 10

[REQUIRED] Test case

Cloud Function responsible for image optimization (based on official extension). Downloads stored file to tmp directory, performs optimization and overwrites the orignal file, copying over the firebaseStorageDownloadTokens

functions/optimizeImage.ts

import * as path from 'path';
import * as os from 'os';
import * as functions from 'firebase-functions';
import * as fs from 'fs';
// Admin SDK initialization
import { admin } from '@/lib/firebase/admin';

const storage = admin.storage();
const mkdirp = require('mkdirp');
const sharp = require('sharp');

export const optimizeImage = functions.storage
  .object()
  .onFinalize(async (object) => {
    if (!object.name || !object.contentType?.startsWith('image/')) {
      return null;
    }

    const parts = object.name.split('/');
    const fileName = parts[parts.length - 1];

    const bucket = storage.bucket(object.bucket);
    const originalFile = bucket.file(object.name);
    const tmpFilePath = path.join(os.tmpdir(), object.name);
    const tmpFolderPath = path.dirname(tmpFilePath);
    const optimizedFilePath = path.join(tmpFolderPath, `optimized-${fileName}`);

    try {
      const metaResponse = await originalFile.getMetadata();
      const metadata = metaResponse?.[0]?.metadata;

     // Print the token for debug purposes
     console.log('Firebase Storage download token:', metadata.firebaseStorageDownloadTokens)

      if (metadata.optimized) {
        return;
      }

      await mkdirp(tmpFolderPath);
      await originalFile.download({
        destination: tmpFilePath,
        validation: false,
      });

      await sharp(tmpFilePath)
        .resize({
          width: 1920,
          withoutEnlargement: true,
        })
        .toFormat('jpg')
        .jpeg({
          quality: 70,
          chromaSubsampling: '4:4:4',
          force: true,
        })
        .toFile(optimizedFilePath);

      // Merge original metadata containing firebaseStorageDownloadTokens
      // with custom metadata
      const optimizedFileMetadata = {
        metadata: {
          ...metadata,
          optimized: true,
        },
      };

      await bucket.upload(optimizedFilePath, {
        destination: originalFile,
        metadata: optimizedFileMetadata,
      });

      await fs.unlinkSync(tmpFilePath);
      await fs.unlinkSync(optimizedFilePath);

      return { success: true };
    } catch (error) {
      return Promise.reject(error);
    }
  });

[REQUIRED] Steps to reproduce

  1. Run the emulators: firebase emulators:start
  2. Upload an image to the Storage (for example using the Storage Emulator UI)
  3. Open the file from Storage Emulator and copy file URL from 403 error in console (related to Unexpected Firebase Cloud Storage Emulator behaviour when objects uploaded using the Admin SDK and the Web SDK #3393) and append
    ?alt=media&token=<token> to the link, replacing token with the token you get from optimizeImage function console.log

Alternatively build the URL yourself:
http://localhost:9199/v0/b/<bucket>/o/<file-path>?alt=media&token=<token>

Attempting to obtain the download url using fileRef.getDownloadURL() results in the following error:
image

The above function works fine and returns the URL when running using production Firebase.

[REQUIRED] Expected behavior

Setting firebaseStorageDownloadTokens should allow you to access the file by providing the ?token=<token> in the resource url. This is used by the official image resize extension and works fine in production (even though it is not officialy documented):

image
image

[REQUIRED] Actual behavior

After overwriting the file or even creating a new file with firebaseStorageDownloadTokens set in the metadata, the file is inaccessible and returns 403 Forbidden

image

[2021-05-22T20:44:15.631Z] [runtime-status] [8668] Trigger "optimizeImage" has been found, beginning invocation! {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] Trigger \"optimizeImage\" has been found, beginning invocation!"}}
i  functions: Beginning execution of "optimizeImage" {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"Beginning execution of \"optimizeImage\""}}
[2021-05-22T20:44:15.632Z] [runtime-status] [8668] triggerDefinition {"regions":["europe-central2"],"eventTrigger":{"resource":"projects/_/buckets/<bucket>.appspot.com","eventType":"google.storage.object.finalize","service":"storage.googleapis.com"},"name":"optimizeImage","entryPoint":"optimizeImage"} {"met
adata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] triggerDefinition {\"regions\":[\"europe-central2\"],\"eventTrigger\":{\"resource\":\"projects/_/buckets/<bucket>.appspot.com\",\"eventType\":\"google.storage.object.finalize\",\"service\":\"storag
e.googleapis.com\"},\"name\":\"optimizeImage\",\"entryPoint\":\"optimizeImage\"}"}}
[2021-05-22T20:44:15.632Z] [runtime-status] [8668] Running optimizeImage in mode BACKGROUND {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] Running optimizeImage in mode BACKGROUND"}}
[2021-05-22T20:44:15.633Z] [runtime-status] [8668] ProcessBackground {"eventId":"1621716254902","timestamp":"2021-05-22T22:44:14.902Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects/_/buckets/<bucket>.appspot.com/objects/danielle-cerullo-CQfNt66t
tZM-unsplash.jpg","type":"storage#object"},"data":{"kind":"#storage#object","name":"danielle-cerullo-CQfNt66ttZM-unsplash.jpg","bucket":"<bucket>.appspot.com","generation":"1621716254898","metageneration":"1","contentType":"image/jpeg","timeCreated":"2021-05-22T22:44:14.897Z","updated":"2021-05-22T22:44:14.
900Z","storageClass":"STANDARD","size":"264694","md5Hash":"HWnJ+ZgmyNHPj9HLQqCXwQ==","etag":"someETag","metadata":{"firebaseStorageDownloadTokens":"c2887239-8b0c-4b47-93e1-974415480b6e","optimized":true},"crc32c":"----6A==","timeStorageClassUpdated":"2021-05-22T22:44:14.897Z","id":"<bucket>.appspot.com/dani
elle-cerullo-CQfNt66ttZM-unsplash.jpg/1621716254898","selfLink":"http://localhost:9199/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg","mediaLink":"http://localhost:9199/download/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg?generation=16
21716254898&alt=media"}} {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] ProcessBackground {\"eventId\":\"1621716254902\",\"timestamp\":\"2021-05-22T22:44:14.902Z\",\"eventType\":\"google.storage.object.finalize\",\"resource\":{\"service\":\"stora
ge.googleapis.com\",\"name\":\"projects/_/buckets/<bucket>.appspot.com/objects/danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"type\":\"storage#object\"},\"data\":{\"kind\":\"#storage#object\",\"name\":\"danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"bucket\":\"<bucket>.appspot.com\",\"generation\":\"16
21716254898\",\"metageneration\":\"1\",\"contentType\":\"image/jpeg\",\"timeCreated\":\"2021-05-22T22:44:14.897Z\",\"updated\":\"2021-05-22T22:44:14.900Z\",\"storageClass\":\"STANDARD\",\"size\":\"264694\",\"md5Hash\":\"HWnJ+ZgmyNHPj9HLQqCXwQ==\",\"etag\":\"someETag\",\"metadata\":{\"firebaseStorageDownloadTokens\
":\"c2887239-8b0c-4b47-93e1-974415480b6e\",\"optimized\":true},\"crc32c\":\"----6A==\",\"timeStorageClassUpdated\":\"2021-05-22T22:44:14.897Z\",\"id\":\"<bucket>.appspot.com/danielle-cerullo-CQfNt66ttZM-unsplash.jpg/1621716254898\",\"selfLink\":\"http://localhost:9199/storage/v1/b/<bucket>.appspot.co
m/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"mediaLink\":\"http://localhost:9199/download/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg?generation=1621716254898&alt=media\"}}"}}
[2021-05-22T20:44:15.634Z] [runtime-status] [8668] RunBackground {"data":{"kind":"#storage#object","name":"danielle-cerullo-CQfNt66ttZM-unsplash.jpg","bucket":"<bucket>.appspot.com","generation":"1621716254898","metageneration":"1","contentType":"image/jpeg","timeCreated":"2021-05-22T22:44:14.897Z","updated
":"2021-05-22T22:44:14.900Z","storageClass":"STANDARD","size":"264694","md5Hash":"HWnJ+ZgmyNHPj9HLQqCXwQ==","etag":"someETag","metadata":{"firebaseStorageDownloadTokens":"c2887239-8b0c-4b47-93e1-974415480b6e","optimized":true},"crc32c":"----6A==","timeStorageClassUpdated":"2021-05-22T22:44:14.897Z","id":"<bucket>
/danielle-cerullo-CQfNt66ttZM-unsplash.jpg/1621716254898","selfLink":"http://localhost:9199/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg","mediaLink":"http://localhost:9199/download/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-uns
plash.jpg?generation=1621716254898&alt=media"},"context":{"eventId":"1621716254902","timestamp":"2021-05-22T22:44:14.902Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects/_/buckets/<bucket>.appspot.com/objects/danielle-cerullo-CQfNt66ttZM-unsplash
.jpg","type":"storage#object"}}} {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] RunBackground {\"data\":{\"kind\":\"#storage#object\",\"name\":\"danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"bucket\":\"<bucket>.appspot.com\",\"generation\"
:\"1621716254898\",\"metageneration\":\"1\",\"contentType\":\"image/jpeg\",\"timeCreated\":\"2021-05-22T22:44:14.897Z\",\"updated\":\"2021-05-22T22:44:14.900Z\",\"storageClass\":\"STANDARD\",\"size\":\"264694\",\"md5Hash\":\"HWnJ+ZgmyNHPj9HLQqCXwQ==\",\"etag\":\"someETag\",\"metadata\":{\"firebaseStorageDownloadTo
kens\":\"c2887239-8b0c-4b47-93e1-974415480b6e\",\"optimized\":true},\"crc32c\":\"----6A==\",\"timeStorageClassUpdated\":\"2021-05-22T22:44:14.897Z\",\"id\":\"<bucket>.appspot.com/danielle-cerullo-CQfNt66ttZM-unsplash.jpg/1621716254898\",\"selfLink\":\"http://localhost:9199/storage/v1/b/<bucket>.appsp
ot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"mediaLink\":\"http://localhost:9199/download/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg?generation=1621716254898&alt=media\"},\"context\":{\"eventId\":\"1621716254902\",\"timestamp\":\"2021-05-22T22:44:14.902Z\",\"even
tType\":\"google.storage.object.finalize\",\"resource\":{\"service\":\"storage.googleapis.com\",\"name\":\"projects/_/buckets/<bucket>.appspot.com/objects/danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"type\":\"storage#object\"}}}"}}
>  Firebase Storage download token: c2887239-8b0c-4b47-93e1-974415480b6e {"user":"Firebase Storage download token: c2887239-8b0c-4b47-93e1-974415480b6e","metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"\u001b[90m> \u001b[39m Firebase Storage download token: c2887239-8b0c-4
b47-93e1-974415480b6e"}}
[2021-05-22T20:44:15.672Z] [runtime-status] [8668] Ephemeral server survived. {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] Ephemeral server survived."}}
i  functions: Finished "optimizeImage" in ~1s {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"Finished \"optimizeImage\" in ~1s"}}
[2021-05-22T20:44:15.675Z] [worker-optimizeImage-70ee1f4d-bb78-4c7e-9017-c7d8108f8742]: IDLE {"metadata":{"emulator":{"name":"functions"},"message":"[worker-optimizeImage-70ee1f4d-bb78-4c7e-9017-c7d8108f8742]: IDLE"}}
[2021-05-22T20:44:15.676Z] [work-queue] {"queueLength":0,"workRunningCount":0}
@abeisgoat
Copy link
Contributor

Interesting, thanks for the report, we probably drop the tokens because it's a special field in the metadata, we probably just missed implementing a setter. Will get it fixed for for a release this week.

abeisgoat added a commit that referenced this issue May 25, 2021
@abeisgoat
Fixes #3396
6714788
@samtstern samtstern mentioned this issue May 26, 2021
Merged
jhengineerartist added a commit to jhengineerartist/art-website that referenced this issue Sep 2, 2023
@jhengineerartist
wip of my first attempt to implement a
60bbf9b 
firebase admin based backend.

I have an issue where the obtained signed URL links to
production google servers rather than the emulated storage,
and calling getDownloadURL to obtain the public url just
straight up crashes the emulator.

Some relevant links to investigate that particular issue.
firebase/firebase-admin-node#1352
firebase/firebase-tools#3396
invertase/react-native-firebase#6638
https://stackoverflow.com/questions/42956250/get-download-url
-from-file-uploaded-with-cloud-functions-for-firebase

This changelist will reproduce the issue if you try to change to
getDownloadURL or try to get an emulated storage url

THIS IS NOT A FUNCTIONAL CHANGELIST.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@abeisgoat abeisgoat

Labels
emulator-suite emulators: storage type: bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Storage Emulator "Fast-Follow" Bug Fixes firebase/firebase-tools
3 participants
@abeisgoat @samtstern @miikebar