有一个需求是实现把python脚本提交为spark任务封装为api，这里思路是用一个flask框架，传递json参数是脚本文件路径和名称，这样api就知道要执行哪个脚本了。关闭也是一样，用json参数传递一个关闭动作，api服务就关闭了。

api.py

from flask import Flask, request, jsonify
import os
import subprocess
import signal
import sys
app = Flask(name)
@app.route('/shutdown', methods=['POST'])
def shutdown():
    if request.method == 'POST':
        shutdown_server()
        return jsonify({"message": "Server is shutting down…"}), 200
def shutdown_server():
    pid = os.getpid()
    os.kill(pid, signal.SIGINT)
@app.route('/submit_spark_job', methods=['POST'])
def submit_spark_job():
    data = request.get_json()
    file_path = data['file_path']
    file_name = data['file_name']
    # 检查文件是否存在
    if not os.path.exists(file_path):
        return jsonify({"error": "文件不存在"}), 400
    # 构建完整的文件路径
    full_file_path = os.path.join(file_path, file_name)
    # 提交Spark任务
    try:
        result = subprocess.run(['/usr/local/spark/bin/spark-submit', full_file_path], capture_output=True, text=True)
        return jsonify({"result": 'Successfully executed Spark task'}), 200
    except Exception as e:
        return jsonify({"error": str(e)}), 500
def signal_handler(sig, frame):
    print('收到关闭信号，正在关闭服务器…')
    sys.exit(0)
if name == 'main':
    signal.signal(signal.SIGINT, signal_handler)
    signal.signal(signal.SIGTERM, signal_handler)
    app.run(host='0.0.0.0', debug=True, port=8084)

运行api.py

在api.py相应目录下执行这个

python3 api.py

查看python进程都有什么

ps -ef | grep python | grep -v grep

post测试

这里我们可以选择burp来进行post测试，当然用postman也可以，关闭命令的post格式如下：

POST /shutdown HTTP/1.1
Host: 192.168.1.110:8084
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.5790.171 Safari/537.36
Accept: application/json
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9
Connection: close
Content-Type: application/json
Content-Length: 21

{
  "action": "shutdown"
}

运行指定脚本的post格式如下：

POST /submit_spark_job HTTP/1.1
Host: 192.168.1.110:8084
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.5790.171 Safari/537.36
Accept: application/json
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9
Connection: close
Content-Type: application/json
Content-Length: 57

{
  "file_path": "/py/",
  "file_name": "example.py"
}