menu

ZEIT

Our mission is to make cloud computing as easy and accessible as mobile computing. You can find our Next.js community here.

Channels
Team

Logflare Zeit Log Integration

February 10, 2020 at 9:50pm

Logflare Zeit Log Integration

February 10, 2020 at 9:50pm

Hiya!! I have a new log integration in beta and would really love some users! Install it here:

https://zeit.co/integrations/logflare


February 10, 2020 at 9:51pm

For Zeit users, we are parsing out the lambda log message into something structured. e.g.

{
"deploymentId": "dpl_6mu1EzaxRH7ocYd87rihaPmt92Ez",
"host": "zeit-app-ht8fw79uu.now.sh",
"id": "1581370267864908883154379412",
"message": "START RequestId: 6f12be7a-6fc4-45f2-9368-8d701552a2d8 Version: $LATEST\nEND RequestId: 6f12be7a-6fc4-45f2-9368-8d701552a2d8\nREPORT RequestId: 6f12be7a-6fc4-45f2-9368-8d701552a2d8\tDuration: 139.77 ms\tBilled Duration: 200 ms\tMemory Size: 1024 MB\tMax Memory Used: 83 MB\t\n",
"parsedLambdaMessage": {
"billed_duration_ms": 200,
"duration_ms": 140,
"max_memory_used_mb": 83,
"memory_size_mb": 1024,
"parse_status": "success"
},
"path": "api/ui.js",
"projectId": "QmZr8SpvHQt5qtCnq4vdPBB3hMwjMmmCwt2jzhyfg9j7xj",
"proxy": {
"clientIp": "184.98.82.152",
"host": "zeit.logflare.app",
"method": "POST",
"path": "/api/ui?nonce=1581370267213",
"region": "lax1",
"scheme": "https",
"statusCode": 200,
"timestamp": 1581370267863,
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36"
},
"requestId": "fh6px-1581370267863-cc5d2faba7e9",
"source": "lambda",
"statusCode": 200,
"timestamp": 1581370267864
}
  • reply
  • like

Notice the parsedLambdaMessage object.

  • reply
  • like

March 23, 2020 at 5:39pm

Hi chase! I've been using this for a bit and had some feedback. I love what you're doing for the zeit logs and formatting them, but for logging i do in my functions (via console.log console.error etc.) I have some issues:

  • logs seem truncated sometimes
  • logs seem batched (ie, if i do 5 console logs in a function there's not a log entry for each one, there's one entry with the log statements bunched together)
  • reply
  • like

these few issues make it hard for me to debug json objects i'm logging or to easily spot error messages. i'd be happy to show you both my app and logs if it deemed helpful. logflare is awesome despite these issues

  • reply
  • like

Zack! Thanks for using Logflare, this feedback is great.

Maybe Zeit is doing the truncating? I'm not totally sure, and haven't seen that. I'll ask a couple people.

Can you email me at support@logflare.app and I'll have a look at your batched log lines? This is true if you do a console.whatever multiple times it will bunch them together but we are trying to parse that into metadata.parsedLambdaMessage.lines where each line should be one of those 5 messages. If that is not working we can probably fix.

  • reply
  • like

Ideally if you're going to invest time into structured logging it would probably be best to write a quick fn to log stuff directly to us and let Zeit handle the exception messages. Or use an exception logger that can POST data to any endpoint.

If you'd like to POST any sort of JSON to us here is a good example:

https://twitter.com/logflare_logs/status/1240622887091851264

Happy to work with you more directly if you email me too. Glad you like Logflare so far!

  • reply
  • like

Thanks for the super speedy reply! Yeah i think the truncating is happening at the zeit level. i peeked in the metadata for a log and saw that the "parsedLambdaMessage" field had the half of a json object with the first ~10 or so lines missing, so the parsing failed. Here's an example: https://gist.github.com/zackkrida/173b948f8f0b9317462cd4f9a56c8cab

i'll probably bypass zeit like you said to get around this, thanks for the help

  • reply
  • like

March 24, 2020 at 2:46pm

Hey Zack, I heard back from the ZEIT team ... their serverless infra sends back the logs truncated, so yeah not much we can do there. Let me know if you'd like help setting up some custom structured logging / exception logging.

  • reply
  • like

thanks chase! i've already switched over to a more robust setup—posting straight to logflare and using pinojs. thanks for helping me break into the world of structured logging!

  • reply
  • like

March 25, 2020 at 9:53pm

Hey

thanks for taking the lead on the official Loglare transport for Pino! If you like to log straight from your Node project this is the perfect way to do it.

  • reply
  • like

March 26, 2020 at 3:03pm

So this integration is really intriguing as Zeit logging is quite lacking. I went to try it out and while adding this statement scared and stopped me: “Logflare” will have complete access to your account.

  • reply
  • like

Our ZEIT app needs access yes (which is different from the Logflare web app). We need to make API requests to create/delete log drains for your ZEIT account. We need access to do that and asfaik ZEIT does not support fine grained access controls yet.

Since we basically used their log drain integration boilerplate template, I've made that repo public if you'd like to look for yourself >> https://github.com/Logflare/zeit-app

  • reply
  • like

March 27, 2020 at 4:22pm

I like this. The ability to analyze the data at data studio has provided me more useful information in the last few hours than I have been able to get to date. So the scary question, how does pricing work as logging services are notoriously wicked expensive (looking at you papertrail).

  • reply
  • like

Thanks! That is one of the main things we're trying to solve actually. The way we do that is by allowing you to "Bring your own backend" (currently BigQuery) so you pay us to pipe data where you want it and then you pay Google/AWS/whoever for the storage and query costs. So basically we just don't mark up the storage. Pricing should land somewhere very close to this.

Edited
like-fill
1
  • reply
  • like

And BigQuery was literally built for this kind of stuff. The data storage is basically equal to S3, and if data is in a partition that hasn't been modified in three months they automatically halve the storage costs (and we use ingest time partitioned tables).

  • reply
  • like

^^^ (not sure if/when you get notifications here)

  • reply
  • like