nooorofe
> Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server?
Webserver is mostly UI. Scheduler service triggers the jobs.
We have groups which run everything as Bash Operator, no dependency issues that way.
You maybe have a very specific use case in mind, the main points of using Airflow for me
* Single orchestration center: manual job control (stop, pause, rerun), backfill; automated scheduler/retry; built-in notification
* Framework built around "reporting period" - it enforces correct abstraction, if a data batch is broken, I can rerun it and rerun all dependent downstream. How do you fix data in event driven workflow?
* managing dependencies
In most cases all Airflow does is running your job with passing it "date" parameter. You can test your code without Airflow - just pass it a date and run from command line.
Airflow has sensors and triggers. https://airflow.apache.org/docs/apache-airflow/stable/author...
But in the core it is built around data pipeline concept, event driven pipeline will much more fragile. Airflow intentionally doesn't manage business logic, it works with "tasks".
You need to define what "latency" means in your case and what is "quite high levels". We are talking about analytical data storage, it is designed for efficient batch processing. To find a single record is not a primary goal of the architecture - you will need some kind of caching/indexing for fast search. Sometimes adding "limit 1" for your single record search may solve the problem.
Be sure you are using efficent data storage format as parquet, check size of the files to be sure you don't have the ["small file problem"](https://www.royalcyber.com/blog/data-services/managing-small...), then check if you are using relevant BigQuery features. And before and after those checks run "explain" on your query, if you don't use partition keys or indexed columns your search results won't be instant in any big data system.
from zoneinfo import ZoneInfo
import pendulum
import datetime
def pendulum_to_datetime(pendulum_dt: pendulum.DateTime) -> datetime.datetime:
return datetime.datetime.fromtimestamp(pendulum_dt.timestamp(),
ZoneInfo(pendulum_dt.timezone_name))
# test
df = pd.DataFrame([[1, 2], [1, 2]], columns=['a', 'b'])
df["time_column"] = pendulum_to_datetime(pendulum.now())
print(df)
a b time_column
0 1 2 2023-11-19 18:14:16.027777-05:00
1 1 2 2023-11-19 18:14:16.027777-05:00
>> df.dtypes
a int64
b int64
time_column datetime64[ns, America/New_York]
dtype: objectIt is best until you work with Pandas DataFrame (https://stackoverflow.com/questions/47849342/making-pandas-w...)
Google library you need "java http client"
https://openjdk.org/groups/net/httpclient/intro.html - it doesn't look complicated.
First example is using "reactive-streams", you don't have to use that syntax if you aren't familiar (same as in Python there are new features, which beginners may learn later), other examples strait forward.
If library you need doesn't exist in standard library, download jar, add it your `classpath` and use.
Skip any tutorials which are using Spring (unless you want to learn Spring). If you want to write more advanced Java code, learn about [Maven](https://maven.apache.org/)
Same thing about JavaEE/JavaSE, if you google about it, you will find that you may use any, there are different libraries included in installation, it doesn't mean you can't add them later.
Periods when engineer don't understand the problem should be spent on analysis of the problem domain. "Now I am working on defining the problem domain" - is an activity to work "I don't understand the problem" task. During that period probably zero code will be written.
> That author only encounters problems that have been fully solved before
He doesn't, otherwise there would be no talk about "plan B" and risks. When you actively write a project code, you should know that solution is possible. Having plan doesn't mean "problems have been fully solved before". You may have POC which doesn't end in resolution, but it should be clear what is POC for and a failure is possible outcome.
* run job for given reporting_period * backfill data for reporting_period between N and M * lookup failed jobs
Nothing in Databricks supports that.
I am not sure what "essential tool" means if pandoc included in it. For me Intellij, Eclipse, Dbeaver are more essential (+ few other Specific tools based on Eclipse).
C# - I am not sure, MS Office?
My brain puts it next to: globe of China, Jewish Olympic games (real thing).
Few things made me confused a bit
* With objects, when I need to use ref and when not.
nim> type
.... Node = ref object
.... le, ri: Node
.... data: int
....
....
nim> var n = Node(data: 9)
nim> n.data
9 == type int
nim> type
.... NodeO = object
.... lw, ri: ref Node
.... data: int
....
....
nim> var no = NodeO(data: 9)
nim> no.data
9 == type int
nim>
That time development/coding wasn't my full time job, Python was the "to go" option of automating some tasks, sometimes Java. In both cases there was no magic in the code - `module.something` easy to track to the origin.
Nim's orientation to standard library is nice. I've started checking Nim when I need a faster alternative for parsing big json and csv (compressed and flat) files. I both cases to was easy to archive, easier than Java (with Java I need to match version and bring dependencies) and the speed was fantastic.
Probably the main reason I haven't continued with Nim, was the company's antivirus at some point started to blocks nimble.exe execution on Windows. Another reason is switching to work into data engineering aria and there is no real need for fast compiled program. IDE support wasn't great too, VScode suggested a lot of irrelevant things, but I think it is kind of broken for any language now (out of box).
import pandas as pd
import numpy as np
if __name__ == "__main__":
n_samples = 10000
samples_np = pd.DataFrame(np.random.randint(1, 7, n_samples), columns=["face_value"])
print(samples_np.face_value.mean())
if __name__ == "__main__":
n_samples = 10000
sample_pd = pd.DataFrame({'face_value': [1, 2, 3, 4, 5, 6]})
print(sample_pd.sample(
n=n_samples,
replace=True,
random_state=np.random.bit_generator.randbits(20)).face_value.mean())
Type annotation, I like the idea, but in the end code looks like Java, but doesn't performs like Java. It is very hard to make it right in Python, also some of them wrong.
( @dataclass(frozen=True): - don't need ":" Gaussian.sample - missing return )
when return added it doesn't return `-> Sequence[float]:`
Gaussian().sample(90).dtype
>>> dtype('float64')
I don't believe "scientific code" is fundamentally different from any other code, I would go with following normal development practices
1) review design ("don't reinvent wheel")
2) add tests
3) make code review
4) version control
etc.
(ex. https://help.talend.com/r/mjoDghHoMPI0yuyZ83a13Q/YYVSsDiyJ3v...)