蚂蚁蚂蚁的技术分享

3D Printing Models Thingiverse 创想云 https://pinshape.com/ https://fab365.net/ Ender3 S1开箱测评

ESP-SR Github - esp-sr 小智 EchoEar 乐鑫 - EchoEar Xiaozhi 飞书文档 - 小智 AI 聊天机器人百科全书 Github - xiaozhi-esp32 Github - xiaozhi-esp32-server Server Firmware Install ESP-IDF ESP-IDF - Linux 和 macOS 平台工具链的标准设置 # Install dependencies brew install cmake ninja dfu-util ccache # clone esp-idf mkdir -p esp && cd esp git clone --recursive https://github.com/espressif/esp-idf.git # install esp-idf cd esp-idf ./install.sh esp32c3 # set environment source export.sh # build cd examples/get-started/hello_world idf.py build # flash to device idf.py -p PORT flash Build mkdir xiaozhi && cd xiaozhi git clone https://github.com/78/xiaozhi-esp32.git # checkout to release version git checkout v2.0.2 # modify config sed -i -e 's\https://api.tenclass.net/xiaozhi/ota/\http://192.168.3.3:8003/ota/\g' main/Kconfig.projbuild API LLM 阿里云百炼 - 模型列表与价格 TTS 火山引擎 - 语音合成大模型 阿里云百炼 - CosyVoice

+++ title = ‘AIGC’ tags = [“AI”] draft = false +++ Image Stable Diffusion text_encoder: Stable Diffusion uses CLIP, but other diffusion models may use other encoders such as BERT tokenizer: must match the one used by the text_encoder model scheduler: the scheduling algorithm used to progressively add noise to the image during training unet: the model used to generate the latent representation of the input vae: autoencoder module that we’ll use to decode latent representations into real images Tutorial Generative Modeling by Estimating Gradients of the Data Distribution The Annotated Diffusion Model Stable Diffusion with Diffusers Install conda create --name=ai python=3.10.9 sudo apt install nvidia-cuda-toolkit pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 pip install transformers noglob pip3 install diffusers["torch"] pip install -U xformers Colab How to Use Stable Diffusion to Generate Images Stable diffusion web-ui References ...

AIOps Concepts XDR（Extended Detection and Response）：综合网络、终端、云等层面 EDR（Endpoint Detection and Response）：主要关注终端层面的威胁和响应 NDR（Network Detection and Response）：主要关注网络层面的威胁和响应 SIEM（Security Information and Event Management） Splunk Elastic SIEM Log Rhythm DXL（Data Exchange Layer）：用于安全产品之间通信的协议 SOC（Security Operations Center） IDS（Intrusion Detection System） HIDS (Host-Based Intrusion Detection System) FIM (File Integrity Monitoring) NIDS (Network-Based IDS) Signature-based IDS (Knowledge-based IDS) Anomaly-based IDS NTA(Network Traffic Analysis) DLP(Data Loss Prevention) EDLP(Endpoint-based DLP) NDLP(Network-based DLP) NAC（Network Access Control）：网络准入控制，确保只有符合条件的设备才能访问 Projects OpenDXL OpenXDR OpenEDR OpenSOC GrayLog OSSIM Security Onion Apache Matron IDS Snort Suricata Wazuh Rules Syntax OSSEC Log monitoring/analysis Zeek (Bro) Samhain Labs OpenDLP Sigma OpenSearch-Using Security Analysis OpenSearch Neural Search Plugin Tutorial MSTIC: msticpy is a library for InfoSec investigation and hunting in Jupyter Notebooks Anomaly Detection Log-based Log Parser Github - logpai - Drain3 Github - logpai - Logparser Projects Github - Log-based Anomaly Detection with Deep Learning: How Far Are We? Github - logpai - loglizer Researchers Chongqing University - Hongyu Zhang Microsoft Research Asia - Shaohan Huang References Github - Anomaly Detection Learning Resources Github - Awesome Log Analysis Github - AIOps handbook Mechine Learning Libraries Github - PyOD Time series Libraries Github - tslearn DLP Document Classification Articles ...

ECS 阿里云挂载数据盘 初始化数据盘（Linux） yum update yum install -y e2fsprogs mkfs -t ext4 /dev/vdb mkdir /data mount /dev/vdb /data sh -c "echo `sudo blkid /dev/vdb | awk '{print \$2}' | sed 's/\"//g'` /data ext4 defaults 0 0 >> /etc/fstab" EMR on ECS Componets Hadoop-Common Yarn port: 8088 OSS-HDFS Hive port: 10000 Hudi Iceberg Paimon Spark3 port: 18080 Gateway 部署 Gateway：使用EMR-CLI自定义部署Gateway环境 通过集群Gateway节点提交作业 emrcli gateway deploy \ --clusterId c-b2a8a74c4d44c537 \ --appNames YARN,HIVE,HUDI,ICEBERG,SPARK3 Serverless EMR DLF VVP Flink SQL 对接 DLF Paimon Catalog Serverless Spark 对接 DLF Paimon Catalog Serverless StarRocks 对接 DLF Paimon Catalog ...

Apisix Install docker install docker run -d --restart unless-stopped \ --network ${DOCKER_NETWORK} \ --name ${DOCKER_APISIX_NAME} \ -p 9080:9080/tcp \ -p 9091:9091/tcp \ -p 9092:9092/tcp \ -p 9180:9180/tcp \ -p 9443:9443/tcp \ -e APISIX_STAND_ALONE=true \ -v ${PWD}/apisix/apisix.yaml:/usr/local/apisix/conf/apisix.yaml:ro \ apache/apisix:${APISIX_VERSION} References APISIX - Getting Started APISIX - Deploy Modes API7 Docs - Configure HTTPS for APISIX APISIX - Centralized Authentication with Apache APISIX and Advanced Tricks APISIX - key-auth Usage Route # create upstream curl "http://127.0.0.1:9180/apisix/admin/upstreams/1" -X PUT -d ' { "type": "roundrobin", "nodes": { "httpbin.org:80": 1 } }' # create route curl "http://127.0.0.1:9180/apisix/admin/routes/1" -X PUT -d ' { "methods": ["GET"], "host": "example.com", "uri": "/anything/*", "upstream_id": "1" }' # route with rate limit curl -i http://127.0.0.1:9180/apisix/admin/routes/1 \ -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d ' { "uri": "/index.html", "plugins": { "limit-count": { "count": 2, "time_window": 60, "rejected_code": 503, "key_type": "var", "key": "remote_addr" } }, "upstream_id": "1" }' Consumer # consumer with rate limit curl http://127.0.0.1:9180/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d ' { "username":"consumer1", "plugins":{ "key-auth":{ "key":"auth-one" }, "limit-count":{ "count":2, "time_window":60, "rejected_code":403, "rejected_msg":"Requests are too many, please try again later or upgrade your subscription plan.", "key":"remote_addr" } } }' # consumer with key auth curl http://127.0.0.1:9180/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d ' { "username": "jack", "plugins": { "key-auth": { "key": "auth-one" } } }'

Argo Workflow References Argo Workflows - User Guide Argo Workflow 教程 Install CLI # Detect OS ARGO_OS="darwin" if [[ uname -s != "Darwin" ]]; then ARGO_OS="linux" fi # Download the binary curl -sLO "https://github.com/argoproj/argo-workflows/releases/download/v3.6.5/argo-$ARGO_OS-amd64.gz" # Unzip gunzip "argo-$ARGO_OS-amd64.gz" # Make binary executable chmod +x "argo-$ARGO_OS-amd64" # Move binary to path mv "./argo-$ARGO_OS-amd64" /usr/local/bin/argo # Test installation argo version CLI argo # with kubeconfig argo --kubeconfig ~/.kube/config list Workflow # list workflows argo list argo list --namespace=argo argo list --running Template argo template list Cluster

PyArrow Install: pip install pyarrow Load Data import pyarrow.parquet as pq table = pq.read_table('/path/to/table') Load data from OSS import ossfs import pyarrow.parquet as pq OSS_ENDPOINT = 'http://oss-cn-wulanchabu-internal.aliyuncs.com' OSS_BUCKET = 'bucket-name' OSS_ACCESS_KEY = '***' OSS_ACCESS_SECRET = '***' fs = ossfs.OSSFileSystem(endpoint=OSS_ENDPOINT, key=OSS_ACCESS_KEY, secret=OSS_ACCESS_SECRET) table = pq.read_table(f'{OSS_BUCKET}/path/to/table', filesystem=fs)

Basic Concepts Cookies 一种用于服务器与浏览器交换数据的方式 服务器响应时通过 Set-Cookie 的 HTTP header 写入浏览器 浏览器每次请求自动在 header 中带上 cookies set cookie 时可以设置过期时间、域名、路径、HttpOnly 等条件，满足条件的请求才会带上该 cookie Session 由于 HTTP 请求是无状态的，需要在多次请求中保持用户状态的话需要额外的数据，这个数据可以是一个随机的字符串（Session ID），这样带有同样 Session ID 的请求就可以被关联为同一个会话 Session ID 一般由服务端生成，并保存在服务端，来对每一次的请求进行关联。与客户端的交互可以用 set cookie 的方式实现 Token 客户端发送请求时携带的凭证，用于表明自己的身份（Authentication） 常见 tokens： 随机字符串：通过服务端生成随机字符串（Session ID），然后通过 Set-Cookie 写入客户端的浏览器作为 token 的形式，每次请求会在 header 中的 cookie 带上该 token/session id JWT：一种按照 JWT 通用协议签名过的信息，base64 解码后为 JSON 格式。由于信息经过了签名，有防篡改的功能。 Online Debugger 示例： { "iss": "example.com", # 表示令牌是由 example.com 这个实体发行的 "sub": "1234567890", # 令牌的主题是用户 ID 为 "1234567890" 的用户 "aud": "https://api.example.com", # 令牌的目标接收方 "exp": 1516239022, # 令牌的过期时间 "nbf": 1516238422, # 令牌的生效时间 "iat": 1516237822, # 令牌的发行时间 # 自定义字段，用于传递用户的权限角色、姓名和电子邮件地址等信息 "role": "admin", "name": "John Doe", # "email": "johndoe@example.com", "custom_data": { "department": "Engineering", "team": "Security" } } OTP (One Time Password) 动态口令，一次有效的验证码机制。最常用的 OTP 为 TOTP（Time-based One-Time Password），通常 30s 一变，服务端和客户端（APP）需提前对齐种子、提前校对时间。在同一时间窗口内，客户端（APP）计算的动态口令（OTP）应与服务端一致，从而通过认证。 ...

Definitions 开环 闭环 在环 回灌 PCAP：网络数据包 PCD文件：点云帧 CAN/CANFD： 总线数据包和车载以太网数据包 Tutorials 知乎 - 自动驾驶学习资料合集 Github - Road-To-Autonomous-Driving Birds-eye-view-Perception To Read Tesla FSD专利全栈解读 如何打造自动驾驶的数据闭环？（上） Data Lake https://www.youtube.com/watch?v=Sguvhvwn8m4&list=PL-gIUf9e9CCtGr_zYdWieJhiqBG_5qSPa https://www.youtube.com/watch?v=MaQQmjtFUK8 https://aws.amazon.com/blogs/architecture/field-notes-building-an-autonomous-driving-and-adas-data-lake-on-aws/ Iceberg https://iceberg.apache.org/spark-quickstart/ AVOps https://learn.microsoft.com/en-us/azure/architecture/example-scenario/automotive/autonomous-vehicle-operations-dataops https://aws.amazon.com/cn/blogs/china/autonomous-driving-data-lake-scene-detection/ https://juicefs.com/zh-cn/blog/user-stories/li-autos-practice-of-migrating-data-from-hdfs-to-juicefs https://cloud.tencent.com/developer/article/2193529 https://www.cnblogs.com/yunqishequ/p/16876242.html https://chejiahao.autohome.com.cn/info/14983089 https://www.slidestalk.com/AWS.User_Group/59655?video https://developer.aliyun.com/article/1103513 https://learn.microsoft.com/zh-cn/azure/architecture/solution-ideas/articles/avops-architecture https://github.com/aws-samples/aws-autonomous-driving-data-lake-ros-bag-visualization-using-rviz https://www.databricks.com/dataaisummit/session/building-data-lakehouse-manage-pbs-autonomous-vehicle-data